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Abstract 

The  U.S.  military  has  placed  a  strong  focus  on  the  importance  of  operating  in  a  joint 
environment,  where  capabilities  and  missions  are  shared  between  service  components. 

Protecting  U.S.  forces  is  a  major  consideration  in  the  joint  environment.  The  Joint  Force 
Protection  Advanced  Security  System  (JFPASS)  architecture  has  been  created  to  fill  a  critical 
gap  in  Joint  Force  Protection  guidance  for  systems  acquisition.  The  systems  engineering  (SE) 
field  has  made  wide  use  of  system  architectures  to  represent  complex  systems.  As  fundamental 
SE  principles  become  more  widespread,  analysis  tools  provide  an  objective  method  for  the 
evaluation  of  the  resulting  architectural  products. 

This  study  used  decision  analysis  to  develop  a  standardized,  yet  adaptable  and  repeatable 
model  to  evaluate  the  capabilities  of  the  JFPASS  for  any  installation  or  facility  belonging  to  the 
United  States  Department  of  Defense  (DoD).  Using  the  Value-Focused  Thinking  (VFT) 
methods,  a  value  hierarchy  was  created  by  consulting  with  subject  matter  experts.  The  resulting 
model,  named  Value -Driven  Enterprise  Architecture  (VDEA)  score,  provides  an  analysis  tool, 
which  enables  DoD  decision-makers  to  use  JFPASS  architecture  products  to  quickly  and  easily 
evaluate  the  value  provided  by  the  system;  VDEA  provides  insight  into  the  overall  quality  and 
capability  of  the  system.  Through  the  scoring  and  sensitivity  analysis  functions,  capability  gaps 
and  potential  improvements  can  be  identified.  Future  studies  in  this  area  will  provide  a  vehicle 
for  rating  not  only  operational  level  systems,  but  also  individual  functional  projects  against  other 
alternatives. 
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VALUE-DRIVEN  ENTERPRISE  ARCHITECTURE  EVALUATION 


FOR  THE 

JOINT  FORCE  PROTECTION  ADVANCED  SECURITY  SYSTEM  (JFPASS) 

Chapter  1.  Introduction 

The  Joint  Force  Protection  Advanced  Security  System  (JFPASS)  has  been  created  to 
solve  the  prominent  problem  in  today’s  military  of  protecting  troops  in  a  joint  environment. 
There  currently  is  not  a  comprehensive  method  to  detennine  both  the  quality  of  architectural 
products  and  of  the  instantiated  system.  The  Value -Driven  Enterprise  Architecture  (VDEA) 
evaluation  tool  was  created  to  fill  this  critical  requirement. 

1.1  General  Background 

Force  protection  has  taken  a  prominent  role  in  today’s  environment,  following  the  attacks 
on  the  World  Trade  Center  and  Pentagon,  the  United  States  military  has  been  deployed  in  a  new 
kind  of  warfare.  Given  the  unprecedented  warfare  tactics  (irregular  warfare)  being  employed  by 
the  enemy,  protecting  personnel  and  assets  is  just  as  important  now  as  it  has  ever  been.  To 
combat  the  threats  facing  the  U.S.  military,  a  new  emphasis  has  been  placed  on  joint  operations 
in  which  joint  warfighting  are  essential  to  the  current  military  culture.  Therefore,  the  U.S. 
military  is  seeking  to  improve  the  trust  and  confidence  between  the  separate  services  and  better 
employ  their  individual  core  competencies  to  accomplish  the  mission  of  the  United  States  more 
effectively  (Office  of  the  Chairman  of  the  Joint  Chiefs  of  Staff  (CJCS),  2007;  Joint  Chiefs  of 
Staff,  2004). 

The  new  joint  environment  has  created  a  need  for  guidance  to  govern  the  combined 
operations  of  the  separate  services  (Office  of  the  CJCS,  2007).  There  is  only  general  guidance 
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that  dictates  the  scope  and  range  of  each  separate  service’s  responsibilities  and  individual  service 
documents  that  dictate  their  specific  Concepts  of  Operations  (CONOPS).  However,  the 
combined  library  of  guidance  documents  lacks  overarching  rules  for  how  joint  operations  will  be 
conducted  and  how  the  individual  services  will  proceed  in  an  environment  where  all  operations 
are  handled  by  a  mix  of  service  capabilities. 

The  systems  engineering  field  has  created  several  tools  to  represent  complex  systems, 
such  as  force  protection  systems.  An  important  tool  within  the  Department  of  Defense  is  system 
architecture.  System  architecture  allows  the  user  to  represent  an  extremely  complex  system 
through  a  series  of  “views”  which  present  the  system  through  a  number  of  perspectives.  These 
architectures  are  used  in  the  acquisition  of  a  system  and  through  its  life-cycle  to  document  its 
development.  Judging  the  quality  of  the  architecture  and  the  systems  that  it  represents,  however, 
is  a  challenge.  Several  models  have  been  created  to  evaluate  different  aspects  of  architecture, 
but  few  focus  on  the  entire  portfolio  with  the  instantiated  system  in  mind.  These  evaluation  tools 
are  generally  based  on  the  existing  system,  as  opposed  to  the  needs  of  the  decision-maker.  This 
effort  combines  the  Operations  Research  field,  with  its  Decision  Analysis  tools,  with  the 
Enterprise  Architecture  field  and  its  Architecture  Evaluation  tools.  Specifically,  Value-Focused 
Thinking  is  used  to  evaluate  Systems  Architecture  at  the  intersection  of  these  ideas.  Figure  1 . 1 
shows  a  VENN  Diagram  of  the  research  area. 
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Value-Focused  Thinking  is  an  objective  decision  analysis  approach  intended  to  overcome 
the  problem  of  multicriteria-decision  making.  It  serves  to  eliminate  assumptions  and  reveal  the 
overarching  values  at  the  base  of  a  particular  decision.  Using  a  set  methodology  such  as  this 
allows  decision-makers  to  ensure  that  they  are  getting  the  end  product  that  they  require  and  are 
expecting  to  receive.  Through  an  established  process  of  identifying  objectives;  developing 
values,  measures,  and  weights;  and  then  applying  functions  to  these  values;  a  detailed  numerical 
analysis  can  be  performed  to  compare  alternatives  or  to  evaluate  a  single  alternative  and  show 
areas  lacking  in  the  important  values  (Chambal,  2001). 

1.2  Specific  Background 

To  address  growing  problems  of  multi-service  coordination  within  the  joint  community,  a 
system  was  proposed  which  would  integrate  force  protection.  The  JFPASS  project  began  when  a 
Joint  Capability  Technology  Demonstration  (JCTD)  was  perfonned  for  the  Joint  Force 
Protection  Advanced  Security  System  (JFPASS)  program  (Rains,  2008).  The  JCTD  is  intended 
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to  demonstrate  the  integration  of  various  components  via  a  combined  command  and  control 
architecture.  This  architecture  will  encompass  the  entire  range  of  Joint  Force  Protection 
functions.  It  is  based  upon  the  joint  operational  concepts  of  Detect,  Assess,  Warn,  Defend  and 
Recover  (DAWDR)  (IUBIP,  2006). 

The  goal  of  the  JFPASS  effort  is  to  develop  an  architecture  that  will  represent  the 
JFPASS  system  and  its  Joint  Capability  Technology  Demonstration  (JCTD)  as  required  by 
JCIDS  (Chairman  of  the  Joint  Chiefs  of  Staff  (CJCS),  2007).  This  research  effort  intends  to 
create  an  evaluation  tool  to  evaluate  the  enterprise  architecture,  based  on  stakeholder  values.  All 
aspects  of  this  project  are  based  on  the  direction  given  by  the  IUBIP  and  centered  on  the 
DAWDR  construct.  Specifically  for  this  effort,  the  Detect,  Assess,  and  Warn  aspects  of 
DAWDR  are  being  investigated  (Rains,  2008). 

1.3  Research  Problem 

There  is  currently  no  method  to  evaluate  the  effectiveness  of  a  force  protection  system 
based  solely  on  architectural  products.  With  the  Air  Force-wide  focus  on  using  architecture  as  a 
documentation  method  and  a  procurement  tracking  system,  an  evaluation  method  is  required  for 
DoD-specific  architectures  and  specifically  in  this  case  for  a  force  protection  system  architecture. 

1.4  Research  Objective  and  Questions 

This  thesis  will  determine  specific  Force  Protection  values  and  evaluate  an  existing 
enterprise  architecture  based  on  these  values.  The  JFPASS  architecture  will  be  evaluated  and  an 
analysis  returned  including  critical  deficiencies  and  required  improvements.  This  research  will 
detennine  an  appropriate  evaluation  method  for  existing  architecture  and  a  way  to  recommend 
future  courses  of  action  based  on  a  set  of  existing  products. 
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The  questions  this  research  effort  will  answer  are:  (1)  How  can  VFT  be  applied  to  an 
evaluation  of  a  set  of  architectural  products?  (2)  What  is  the  resulting  value  hierarchy  to 
evaluate  a  force  protection  system?  (3)  What  are  the  related  weights  and  measures  for  the 
hierarchy?  (4)  How  well  does  the  provided  architecture  score  based  on  this  hierarchy  and  where 
are  the  shortfalls  and  potential  areas  of  improvement? 

1.5  Methodology 

Value-Focused  Thinking  will  be  applied  to  architecture  to  evaluate  its  viability.  To  date, 
VFT  has  not  been  used  to  evaluate  an  architectural  product.  In  fact,  there  is  very  little  research 
on  the  topic  of  evaluating  architecture  and  little  to  no  peer-reviewed  research  regarding  the 
analysis  of  a  force  protection  system.  Leveraging  VFT,  a  methodology  for  architecture 
evaluation  will  be  developed. 

1.6  Scope 

The  scope  of  this  thesis  will  be  limited  to  the  architectural  products  and  the  environment 
within  which  these  products  were  intended  to  function.  An  extendable  and  defensible  tool  will 
be  created  to  evaluate  a  set  of  static  architectural  products.  The  scope  of  the  force  protection 
environment  includes  worldwide  military  installations.  It  is  limited,  however,  to  the  realm  of 
joint  operations.  Therefore,  battlespaces  controlled  by  an  individual  service  will  not  be 
addressed.  For  example,  portside  security  will  be  addressed,  but  force  protection  at  sea  is  not 
considered  as  this  is  a  Navy-specific  battlespace.  Space  assets  will  also  not  be  included.  Threats 
from  the  air  will  be  taken  into  consideration,  but  the  airspace  operating  environment  will  not  be 
included.  Since  the  Air  Force  maintains  primary  control  over  air  space  engagements  (although 
all  services  operate  within  this  environment),  the  protection  of  air  assets  is  not  a  joint  operation. 
Within  the  force  protection  area,  the  Detect,  Assess,  Warn,  Defend  Recover  (DAWDR)  construct 
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will  be  included  in  the  evaluation,  although  the  Defend  and  Recover  tenets  are  not  of  primary 


concern. 

1.7  Review  of  Chapters/Research  Approach 

Chapter  2  consists  of  a  review  of  the  available  force  protection/facility  protection 
material,  as  well  as  the  DoD  guidance  governing  the  individual  service’s  force  protection  efforts. 
It  discusses  how  these  documents  relate  to  the  research  effort.  Chapter  3  details  the  methodology 
used  for  this  effort.  Specifically,  it  discusses  the  1 0-step  VFT  process  employed  here  and  how 
each  step  was  used  to  create  the  resulting  hierarchy,  assign  weights,  create  Single  Dimension 
Value  Functions  (SDVF),  and  analyze  the  model.  It  also  discusses  the  collection  of  relevant 
materials  and  communication  with  the  decision  making  entity.  Chapter  4  provides  an  actual 
evaluation  of  the  architecture  in  question.  It  will  show  how  the  instantiated  system  architecture 
scored  on  the  hierarchy  and  areas  of  improvement  to  produce  a  fully  functional  and  effective 
force  protection  system.  Chapter  5  discusses  these  findings  and  their  applicability  to  the  force 
protection  mission.  Chapter  5  also  highlights  the  impact  of  this  effort  and  details  the  future 
research  required  to  continue  this  effort,  as  well  as  how  it  can  be  applied  to  other  areas. 
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Chapter  2.  Literature  Review 


This  chapter  presents  important  previous  research  pertinent  to  this  effort.  Joint  force 
protection  is  presented  to  provide  the  context  for  the  overall  Joint  Force  Protection  Advanced 
Security  System  (JFPASS)  project.  The  field  of  systems  engineering  is  discussed,  as  system 
architecture  is  the  tool  used  to  produce  the  product  being  examined.  Decision  analysis  and 
Value-Focused  Thinking  were  used  to  evaluate  the  provided  architecture.  The  basis  of  the  value 
generation  step  within  Value-Focused  Thinking  was  the  affinity  diagramming  method,  which  is 
taken  from  the  management  and  planning  toolbox.  Finally,  net-centricity  will  be  summarized  as 
it  applies  to  this  project  and  its  impact  on  the  Department  of  Defense  (DoD). 

2.1  Joint  Force  Protection 

The  tenn  Joint  Force  Protection  (JFP)  is  used  by  the  DoD  to  describe  efforts  related  to 

protecting  personnel,  assets,  and  infonnation  among  all  service  components.  Currently,  each 

service  has  its  own  tactics,  techniques,  and  procedures  (TTPs)  for  accomplishing  this  goal. 

Recent  developments  on  the  political  world  stage  have  caused  the  DoD  to  move  toward  a  more 

joint  environment,  as  opposed  to  the  separate  TTPs  fonnerly  used  by  the  services.  This  idea  is 

outlined  in  the  National  Military  Strategy,  which  states  that  “achieving  the  objectives  of  protect, 

prevent,  prevail  requires  connected  joint  operating  concepts  (JOCs)”  (Joint  Chiefs  of  Staff, 

2004).  Furthermore,  Joint  Document  1-02,  the  Department  of  Defense  Dictionary  of  Military 

and  Associated  Tenns,  defines  force  protection  as, 

Preventive  measures  taken  to  mitigate  hostile  actions  against  Department  of 
Defense  personnel  (to  include  family  members),  resources,  facilities,  and  critical 
information.  Force  protection  does  not  include  actions  to  defeat  the  enemy  or 
protect  against  accidents,  weather,  or  disease.  (Joint  Chiefs  of  Staff,  2008,  p.  214) 
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The  Protection  Joint  Functional  Concept  (PJFC)  (Department  of  Defense,  2004)  goes  on 
to  state  that  the  actions  involved  in  force  protection  (FP)  are  intended  to  conserve  the  force’s 
lighting  potential  so  that  it  may  be  applied  at  the  appropriate  time  to  accomplish  the  mission  at 
hand  (Department  of  Defense,  2004).  The  U.S.  definition  also  aligns  very  closely  with  the  North 
Atlantic  Treaty  Organization  (NATO)  definition  of  force  protection  (NATO  Standardization 
Agency,  2008).  This  connection  allows  better  communication  among  multi-national  forces. 

The  military’s  current  method  of  assessing  FP  in  a  facility  is  through  the  use  of  Joint 
Staff  Integrated  Vulnerability  Assessment  (JSIVA)  teams.  These  teams  of  force  protection 
experts  visit  military  installations  to  detennine  their  level  of  protection.  As  part  of  this 
assessment,  they  provide  the  required  training  and  feedback  to  enhance  protection  postures  at 
these  installations.  The  JSIVA  program  provides  a  comprehensive  assessment  tool  for 
operational  facilities,  but  they  have  no  method  for  evaluating  FP  systems  under  design  (Cirafici, 
2002). 

2.1.1  National  Policy 

Joint  force  protection  concepts  are  drawn  from  the  national  strategic  objective.  National 
guidance  regarding  force  protection  and  military  operations  come  in  several  tiers.  The  National 
Security  Strategy  (NSS)  is  the  Presidential  directive  which  guides  all  efforts  to  secure  and  defend 
the  United  States.  It  discusses  international  strategy  as  well  as  the  United  States’  goal  of 
improving  the  quality  of  life  not  only  within  the  U.S.,  but  in  other  countries  as  well.  It  also 
discusses  the  strategic  objective  of  eliminating  terrorism  by  winning  the  War  on  Terrorism 
(Office  of  the  President  of  the  United  States  of  America,  2002). 

The  National  Defense  Strategy  (NDS)  directly  supports  the  NSS  by  establishing 
objectives  by  which  the  goals  of  the  NSS  will  be  accomplished  and  measured.  The  NDS 


provides  the  link  between  the  DoD  and  other  government  agencies  as  they  relate  to  the  security 
objectives  of  the  nation.  The  objectives  set  forth  by  the  NDS  are  to:  (1)  secure  the  United  States 
from  direct  attack,  (2)  secure  strategic  access  and  retain  global  freedom  of  action,  (3)  establish 
security  conditions  conductive  to  a  favorable  international  order,  and  (4)  strengthen  alliances  and 
partnerships  to  contend  with  common  challenges  (Office  of  the  Secretary  of  Defense,  2008). 
These  objectives  provide  the  direction  for  the  National  Military  Strategy  (Office  of  the  Secretary 
of  Defense,  2008). 

The  National  Military  Strategy  (NMS)  provides  the  focus  for  military  activities  by 
specifying  the  overall  objectives  set  forth  in  the  NSS  and  NDS.  To  this  end,  the  NMS  refers 
specifically  to  three  guiding  ideas.  “Protect  the  United  States”  refers  specifically  to  what  has 
become  known  as  “Homeland  Security.”  The  NMS  establishes  homeland  security  as  the  first 
priority  of  the  United  States.  The  armed  forces  are  responsible  for  securing  the  nation,  both  at 
home  and  abroad.  The  military  accomplishes  missions  outside  the  U.S.  to  counter  threats  as  they 
occur  at  their  source.  They  must  then  secure  strategic  approaches  to  the  U.S.  to  ensure  enemy 
forces  cannot  gain  direct  access  to  the  country.  Lastly,  they  must  employ  force  as  directed  on 
home  soil  in  the  case  of  direct  attack.  “Prevent  conflict  and  surprise  attack”  is  the  second  idea 
specified  in  the  NMS.  This  refers  mainly  to  strengthening  alliances  and  creating  a  security 
environment  in  which  aggressions  from  adversaries  is  discouraged.  Preventing  this  conflict  is  a 
goal  which  requires  global  action  and  attention  to  any  adversary  who  may  pose  a  threat  to  the 
United  States.  “Prevail  against  adversaries”  is  the  objective  that  refers  specifically  to  the 
military’s  mission  of  swiftly  defeating  adversaries  in  campaigns  and  wars.  This  objective 
includes  the  ability  to  integrate  all  available  technologies,  capabilities,  and  information  in 
overlapping  campaigns  (Joint  Chiefs  of  Staff,  2004). 
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Accomplishing  the  objectives  of  protect,  prevent,  and  prevail  requires  the  use  of  Joint 

Operational  Concepts  (JOCs).  The  NMS  focuses  largely  on  the  concept  of  a  Joint  Service.  The 

desired  attributes  of  a  joint  force  are  one  that  is:  fully  integrated;  expeditionary;  networked, 

decentralized,  and  adaptable;  has  decision  superiority;  and  is  capable  of  lethality.  The  scope  of 

security  for  the  joint  force  is  defined  in  the  NMS  as: 

the  ability  to  operate  across  the  air,  land,  sea,  space  and  cyberspace  domains  of 
the  battlespace.  Armed  Forces  must  employ  military  capabilities  to  ensure 
access  to  these  domains  to  protect  the  Nation,  forces  in  the  field  and  U.S. 
global  interests.  The  non-linear  nature  of  the  current  security  environment 
requires  multi-layered  active  and  passive  measures  to  counter  numerous 
diverse  conventional  and  asymmetric  threats.  These  include  conventional 
weapons,  ballistic  and  cruise  missiles  and  WMD/E.  They  also  include  threats 
in  cyberspace  aimed  at  networks  and  data  critical  to  U.S.  infonnation-enabled 
systems.  Such  threats  require  a  comprehensive  concept  of  deterrence 
encompassing  traditional  adversaries,  terrorist  networks  and  rogue  states  able 
to  employ  any  range  of  capabilities.  (Joint  Chiefs  of  Staff,  2004,  p.  18) 

By  defining  joint  force  security,  the  National  Military  Strategy  provides  the  context  for  joint 

force  protection.  It  establishes  a  focus  on  joint  operations  and  on  directing  the  actions  of  the 

military.  The  NMS  implies  a  need  to  protect  those  who  are  accomplishing  the  mission.  This 

implication  is  explored  in  more  depth  in  the  implementation  of  the  objectives  set  forth  in  the 

NMS  by  joint  guidance  documents. 

2.1.2  Joint  Guidance  Documents 

In  addition  to  national  policy,  several  joint  documents  have  been  created  to  help  guide  the 
development  of  the  joint  force.  Each  of  these  documents  is  targeted  toward  a  specific  audience 
for  a  specific  purpose.  There  is  overlap  to  each  of  them,  but  their  guidance  is  standard  across  the 
documents.  The  recurring  theme  is  that  the  service  components  must  learn  to  operate  effectively 
in  a  joint  environment.  Each  of  the  following  documents  gives  information  critical  to  operating 
jointly. 
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2. 1.2.1  Joint  Publication  1 


Joint  Publication  1  (JP1)  is  the  overarching  guidance  for  all  other  joint  publications.  It 
provides  the  guidance  for  the  unified  operations  of  all  branches  of  service  by  bridging  policy  and 
doctrine.  This  common  perspective  is  employed  by  all  commanders  to  ensure  that  each  service 
component  is  working  toward  the  same  goal.  This  document  directs  the  services  to  operate 
jointly  by  relying  on  each  other’s  skills  and  capabilities.  JP1  states  that  despite  the  U.S. 
military’s  ability  to  conduct  warfare,  the  military  must  also  focus  on  the  strategic  security 
environment  to  ensure  the  viability  of  its  warfighting  capability  (Office  of  the  CJCS,  2007). 

2. 1.2.2  Joint  Publication  3-0 

Joint  Publication  3-0  (JP3)  extends  the  guidance  in  JP1  to  include  planning  and  execution 
across  the  range  of  military  operations  typically  found  in  the  joint  environment.  In  JP3, 
protection  is  included  as  a  critical  joint  function.  Four  primary  protection  functions  are  outlined 
as  active  defensive  measures,  passive  defensive  measures,  applying  technology  and  procedures, 
and  emergency  management  and  response  (Office  of  the  CJCS,  2008).  JP3  also  extends  force 
protection  to  include  friendly  nations  and  other  allied  organizations.  It  also  discusses  health 
protection  as  a  subsection  of  FP. 

JP3  states  that  the  protection  function  itself  includes  several  tasks.  Each  task  directly 
relates  to  the  Detect,  Assess,  Warn,  Defend,  Recover  (DAWDR)  construct  and  concept  of 
protecting  personnel,  assets,  and  infonnation.  Air,  space,  and  missile  defense;  protection  of 
noncombatants;  physical  security;  antiterrorism;  and  eight  other  tasks  comprise  the  protection 
function  (Office  of  the  CJCS,  2008).  These  protection  tasks  show  the  full  range  of  force 
protection. 
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2. 1.2.3  Protection  Joint  Functional  Concept 

The  Protection  Joint  Functional  Concept  (PJFC)  is  a  DoD  governing  document  regarding 
protection  of  friendly  personnel,  information,  and  assets.  It  is  intended  to  guide  future  joint 
operations  within  all  service  components.  The  PJFC  defines  protection  as  “the  ability  to  sense 
adversary  activities,  understand  their  impact  on  Joint  Force  operations,  and  make  timely  and 
appropriate  decision  to  execute  capabilities  to  neutralize  or  mitigate  adversary  effects” 
(Department  of  Defense,  2004).  This  document  identifies  the  three  key  areas  for  protection: 
Personnel,  Assets,  and  Information.  They  are  defined  by  the  protection  construct  shown  in 
Figure  2.1. 
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Figure  2.1.  The  Protection  Construct  (Department  of  Defense,  2004) 


This  construct  defines  the  five  key  aspects  of  force  protection:  Detect,  Assess,  Warn, 
Defend,  and  Recover  (DAWDR).  The  joint  force  commander  must  be  able  to  effectively  execute 
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each  of  the  DAWDR  tenets  in  a  joint  environment.  The  PJFC  also  provides  the  context  and 
overall  guidance  for  each  of  the  other  Joint  Functional  Concepts,  such  as  Battlespace  Awareness, 
Command  and  Control,  and  Force  Application.  It  also  addresses  the  Mission  Capability  Areas 
(MCAs)  and  Mission  Capability  Elements  (MCEs),  which  are  the  specific  protection  tasks  which 
enable  the  joint  force  to  execute  its  mission  (Department  of  Defense,  2004).  The  hierarchy 
described  in  Figure  2.1  provides  the  context  for  joint  force  protection.  Specifically,  it  defines  the 
scope  of  joint  force  protection  as  falling  within  the  DAWDR  construct.  It  then  specifies 
DAWDR  to  include  personnel,  assets,  and  information,  providing  a  more  specific  scope  for  the 
objective  of  force  protection. 

2. 1.2.4.  Universal  Joint  Task  List  (UJTL) 

The  Universal  Joint  Task  List  (UJTL)  provides  the  standardization  required  to  plan, 
conduct,  evaluate,  and  assess  joint  and  multinational  training.  It  dictates  what  specific  functions 
must  be  accomplished  by  the  joint  force.  When  combined  with  the  Service  Task  Lists,  it 
provides  a  comprehensive  list  of  tasks  and  measures  for  all  levels  of  the  DoD.  The  UJTL  also 
provides  the  context  for  interoperability;  however,  it  does  not  define  how  services  are  expected 
to  interact  with  each  other  in  the  execution  of  the  joint  mission.  The  tasks  are  divided  into 
Strategic  National,  Strategic  Theater,  Operational,  and  Tactical  tasks.  Each  of  these  tasks 
includes  lists  of  subtasks  that  fall  under  the  major  idea.  Under  tactical  tasks,  subsection  six 
focuses  specifically  on  force  protection.  The  operational  context  for  each  task  is  defined  by  the 
joint  conditions  section.  Joint  Condition  2.7  is  the  section  which  focuses  on  the  protection  of 
each  area  of  air,  sea,  and  land.  The  most  critical  part  of  the  UJTL  to  this  study  is  the  definition 
of  the  measures  associated  with  each  protection  area.  For  example,  the  measures  of  Air 
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Superiority  are  defined  as  Full,  General,  Local,  or  No.  The  UTJL,  however,  only  provides  broad 
guidance  and  does  not  dictate  how  each  service  will  fulfill  its  mission  (Joint  Staff,  2002). 

2.1.3  Service  Policies 

Under  the  joint  guidance,  each  individual  service  component  must  create  its  own  force 
protection  guidance  to  dictate  how  the  principles  set  forth  in  joint  and  national  doctrine  will  be 
accomplished.  Military  installations  are  controlled  by  the  owning  service  component;  however, 
in  the  case  of  joint  bases  or  shared  installations,  a  single  service  is  chosen  as  the  lead  for  force 
protection  on  that  installation.  This  presents  problems  because  of  the  different  implementations 
of  the  joint  guidance.  Each  service  operates  within  the  guidelines  set  forth,  but  executes  those 
guidelines  differently. 

2. 1.3.1  Air  Force 

The  Air  Force’s  Installation  Security  Program  (ISP)  is  their  primary  guidance  for  force 
protection.  Air  Force  Instruction  (AFI)  31-101  provides  the  implementation  of  Air  Force  Policy 
Directive  31-1,  Physical  Security.  The  Air  Force  places  primary  responsibility  for  force 
protection  within  their  Security  Forces  career  field.  This  single  career  field  is  responsible  for 
creating  programs  and  regulations  to  ensure  that  the  entire  population  of  the  installation  is 
secure.  Security  Forces  are  responsible  not  only  for  the  implementation  of  the  Air  Force  ISP,  but 
they  are  also  responsible  for  ensuring  that  the  installation  complies  with  each  level  of  guidance. 
They  must  create  and  maintain  an  ISP,  as  well  as  host  the  Installation  Security  Council  (ISC) 

(HQ  AFSFC/SFON  &  SFOP,  2003).  A  portion  of  the  Air  Force  force  protection  responsibility 
falls  to  the  Civil  Engineer  career  field,  as  they  are  responsible  for  designing  and  building  both 
home  station  and  expeditionary  structures  which  must  comply  with  the  Anti-Terrorism/Force 
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Protection  (ATFP)  guidelines  as  well  as  the  doctrine  set  forth  in  joint  publications  and 
installation-specific  regulations. 

2. 1.4.2  Army 

The  Anny’s  physical  security  program  is  a  component  of  its  force  protection  program. 

The  Anny’s  program  relies  on  the  military  police  force,  but  it  also  makes  use  of  all  other  soldiers 
to  implement  the  policies  and  guidance.  For  example,  physical  security  inspectors  can  be  from 
any  military  Occupational  Specialty  (Department  of  the  Anny,  1993).  This  policy  makes  force 
protection  a  more  implicit  responsibility.  The  regulations  within  the  Anny  are  carried  out  by 
programs  put  in  place  at  higher  headquarters,  but  compliance  is  a  command  responsibility. 

2. 1.3.3  Navy/Marine  Corps 

The  Navy  and  Marine  Corps  have  an  entirely  different  approach  to  force  protection. 

Since  they  spend  the  majority  of  their  time  at  sea,  there  is  a  command  within  the  Navy  known  as 
the  Force  Protection  Command.  Its  primary  duty  is  to  protect  Naval  forces  from  Naval  threats. 
The  Navy’s  port  security  program  is  managed  either  by  civilians  or  their  ship  security  personnel 
(NTTP  3-07.2.1,  2003;  NWP  3-07.2  (Rev  A),  2004). 

2.1.4  Integrated  Unit  Base  Installation  Protection  (IUBIP) 

Integrated  Unit  Base  Installation  Protection  (IUBIP)  has  three  guiding  documents:  the 
IUBIP  Concept  of  Operations  (CONOPs),  the  IUBIP  Functional  Area  Analysis  (FAA),  and  the 
IUBIP  Joint  Capability  Document  (JCD).  The  IUBIP  CONOPs  “conceptualizes  the  integration 
of  protection  capabilities  for  agile,  decisive,  and  integrated  force  employment  in  all  phases  of 
combat  and  supporting  operations”  (IUBIP,  2006).  The  IUBIP  FAA  defines  the  tasks  required  of 
the  joint  force  to  accomplish  the  goal  of  protecting  personnel,  information,  and  assets  (Joint 
Chiefs  of  Staff,  2007).  The  IUBIP  JCD  discusses  the  Joint  Functional  Areas,  as  well  as  the 
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required  capabilities,  capability  gaps,  and  threat  environment  (Joint  Requirements  Oversight 
Council,  2007). 

These  documents  provide  the  context  for  the  specific  project.  The  IUBIP  CONOPs, 
which  builds  upon  the  FAA  and  JCD,  defines  the  military  problem  for  which  the  Joint  Force 
Protection  Advanced  Security  System  (JFPASS)  is  being  created.  Specifically,  as  the  joint  force 
is  reduced  due  to  budgetary  constraints,  it  will  require  more  efficiency  to  continue  its  current 
level  of  operations.  The  IUBIP  states  that  the  joint  force  has  a  great  deal  to  gain  from  integration 
and  explains  how  this  may  be  accomplished.  The  CONOPs  discuss  the  benefit  that  net-centricity 
will  have  on  a  newer,  integrated  joint  force  and  how  net-centricity  is  required  as  the  joint  force 
matures  (IUBIP,  2006).  Net-centricity  is  typically  defined  as  the  operation  of  a  group  of  nodes 
in  communication  with  each  other. 

2.2  Systems  Architecture 

With  the  complexity  of  force  protection,  a  system  of  analysis  is  required  to  gain  an 
understanding  of  how  the  individual  services  interrelate.  The  systems  engineering  field  is  an 
interdisciplinary  approach  to  enable  the  realization  of  successful  systems  (Blanchard  & 

Fabrycky,  2006).  The  creation  and  design  of  these  systems  is  accomplished  through  graphical 
representations,  design  tools,  and  process  analysis.  Through  the  use  of  these  tools  and  system 
analysis,  system  designers  and  managers  can  better  understand  and  therefore  manage  their 
systems. 

One  of  the  tools  within  the  systems  engineering  field  that  has  gained  wide  use  and 
acceptance  within  the  DoD  is  systems  architecture.  The  DoD  Architecture  Framework  (DoDAF) 
was  written  as  the  authoritative  source  on  DoD’s  use  and  implementation  of  architecture.  The 
DoDAF  describes  architecture  as  “the  structure  of  components,  their  relationships,  and  the 


16 


principles  and  guidelines  governing  their  design  and  evolution  over  time”  (DoD  Architecture 
Framework  Working  Group,  2007).  Maier  and  Rechtin  (2002)  refer  to  architecture  simply  as 
“the  art  and  science  of  designing  and  building  systems.”  The  DoDAF  prescribes  a  systematic 
process  of  architecture  by  providing  the  standards  by  which  architecture  “views”  or  products 
(discussed  in  Section  2.2.1)  are  created,  while  Maier  and  Rechtin  (2002)  tend  to  believe  that 
architecture  is  a  more  abstract  concept  which  requires  a  “process  of  insights,  vision,  intuitions, 
judgment  calls,  and  even  taste.” 

2.2.1  DoDAF 

The  DoDAF  is  the  result  of  at  least  12  years  of  evolution  of  DoD  policy  and  procedures 
(DoD  Architecture  Framework  Working  Group,  2007).  It  began  with  the  Command,  Control, 
Communications,  Computers,  Intelligence,  Surveillance  and  Reconnaissance  (C4ISR) 
Architecture  Framework  vl.O  in  1996  followed  by  v2.0  in  1997.  The  DoDAF  vl.O  was 
subsequently  released  in  2003.  The  current  version  of  DoDAF  is  1.5  and  was  released  in  2007. 
DoDAF  v2.0  has  been  released  in  draft  form  and  is  being  coordinated  for  an  official  release 
expected  by  the  middle  of  2009.  All  of  these  efforts  are  the  result  of  the  move  toward  joint  and 
multinational  operations  (DoD  Architecture  Framework  Working  Group,  2007).  The  DoD  has 
ensured  the  use  of  DoDAF  through  the  use  of  policies  and  directives  which  require  its  use  in 
acquisition  processes.  Table  2.1  displays  the  evolution  of  the  policy  documents  that  have 
directed  the  use  of  DoDAF. 
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Table  2. 1 .  Federal  Policy  for  Architectures  (DoD  Architecture  Framework  Working 
_  Group,  2007) _ 


Policy/Guidance 

Description 

Clinger-Cohen  Act  of 

1996 

Recognizes  the  need  for  Federal  Agencies  to  improve  the 
way  they  select  and  manage  IT  resources  and  states 
information  technology  architecture,  with  respect  to  an 
executive  agency,  means  an  integrated  framework  for 
evolving  or  maintaining  existing  IT  and  acquiring  new  IT  to 
achieve  the  agency’s  strategic  goals  and  information 
resources  management  goals”.  Chief  Information  Officers 
are  assigned  the  responsibility  for  “developing,  maintaining, 
and  facilitating  the  implementation  of  a  sound  and  integrated 
IT  architecture  for  the  executive  agency.” 

Office  of  Management  and  Budget 
Circular  A- 130 

“Establishes  policy  for  the  management  of  Federal 
information  resources”  and  calls  for  the  use  of  Enterprise 
Architectures  to  support  capital  planning  and  investment 
control  processes.  Includes  implementation  principles  and 
guidelines  for  creating  and  maintaining  Enterprise 
Architectures. 

E-Govemment  Act  of  2002 

Calls  for  the  development  of  Enterprise  Architecture  to  aid  in 
enhancing  the  management  and  promotion  of  electronic 
government  services  and  processes. 

OMB  Federal  Enterprise 
Architecture  Reference  Models 
(FEA  RM) 

Facilitates  cross-agency  analysis  and  the  identification  of 
duplicative  investments,  gaps,  and  opportunities  for 
collaboration  within  and  across  Federal  Agencies.  Alignment 
with  the  reference  models  ensures  that  important  elements  of 
the  FEA  are  described  in  a  common  and  consistent  way.  The 
DoD  Enterprise  Architecture  Reference  Models  are  aligned 
with  the  FEA  RM. 

OMB  Enterprise  Architecture 
Assessment  Framework 
(EAAF) 

Serves  as  the  basis  for  enterprise  architecture  maturity 
assessments.  Compliance  with  the  EAAF  ensures  that 
enterprise  architectures  are  advanced  and  appropriately 
developed  to  improve  the  performance  of  information 
resource  management  and  IT  investment  decision  making. 

General  Accounting  Office 
Enterprise  Architecture 

Management  Maturity  Framework 
(EAMMF) 

“Outlines  the  steps  toward  achieving  a  stable  and  mature 
process  for  managing  the  development,  maintenance,  and 
implementation  of  enterprise  architecture.”  Using  the 

EAMMF  allows  managers  to  determine  what  steps  are 
needed  for  improving  architecture  management. 
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There  are  several  types  of  architecture  in  systems  engineering  resources.  DoDAF 
discusses  Integrated  Architectures,  Composite  Architectures,  and  Federated  Architectures. 
Integrated  architectures  are  those  in  which  there  is  concordance  between  all  products  and 
entities.  They  use  a  standard  nomenclature  throughout  the  operational  views  (OV),  systems  and 
services  views  (SV),  all  views  (AV),  and  technical  views  (TV).  In  this  case,  operational  views 
are  those  which  describe  the  general  tasks,  activities,  and  major  information  exchanges.  Systems 
and  services  views  capture  specific  interconnection  information  further  specifying  the 
information  found  in  OVs.  Technical  views  contain  the  minimum  set  of  rules  which  govern  the 
functions  of  the  system  or  system  elements.  All-views  are  the  overarching  informational  views. 
They  provide  information  about  the  architecture,  but  do  not  actually  show  an  architectural  view. 
Table  2.2  shows  all  views  included  within  DoDAF.  Integrated  architectures  facilitate  ease  of  use 
and  communication,  as  well  as  aggregation  of  infonnation.  Composite  architectures  are  those 
composed  of  separate  parts.  Generally,  several  integrated  architectures  are  pulled  together  to 
form  composite  architectures  which  support  a  more  broad  set  of  goals.  Finally,  federated 
architectures  are  distributed  infonnation  bases  compiling  infonnation  of  use  to  decision-makers 
at  higher  levels.  All  architectures  increase  the  net-centricity  of  a  system  by  encouraging  the 
process  of  examining  links  between  nodes  and  modeling  the  composition  of  the  system. 

The  DoDAF’s  use  has  become  commonplace  within  several  areas  of  the  DoD, 
particularly  within  the  acquisition  process.  The  Joint  Capabilities  Integration  and  Development 
System  (JCIDS)  formally  defines  the  acquisition  process  and  directs  the  use  of  certain 
architecture  products  for  milestone  decision  points  (CJCS,  2007).  For  example,  Milestone 
Decision  Point  A  requires  an  OV-1  view  for  consideration  of  the  project  (CJCS,  2007). 
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Table  2.2  Do! 

DAF  Views  (DoD  Architecture  Framework  Working  Group,  2007) 

Applicable  View 

Framework 

Product 

Framework  Product  Name 

All  View 

AV-1 

Overview  and  Summary  Information 

All  View 

AV-2 

Integrated  Dictionary 

Operational 

OV-1 

High-Level  Operational  Concept  Graphic 

Operational 

OV-2 

Operational  Node  Connectivity  Description 

Operational 

OV-3 

Operational  Information  Exchange  Matrix 

Operational 

OVA 

Organizational  Relationships  Chart 

Operational 

OV-5 

Operational  Activity  Model 

Operational 

OV-6a 

Operational  Rules  Model 

Operational 

OV-6b 

Operational  State  Transition  Description 

Operational 

OV-6c 

Operational  Event-Trace  Description 

Operational 

OV-7 

Logical  Data  Model 

Systems  and  Services 

SV-1 

Systems/Services  Interface  Description 

Systems  and  Services 

SV-2 

Systems/Services  Communications  Description 

Systems  and  Services 

SV-3 

Systems/Services-Systems/Services  Matrix 

Systems  and  Services 

SV-4a 

Systems  Functionality  Description 

Systems  and  Services 

SV-4b 

Services  Functionality  Description 

Systems  and  Services 

SV-5a 

Operational  Activity  to  System  Function  Traceability  Matrix 

Systems  and  Services 

SV-5b 

Operational  Activity  to  System  Traceability  Matrix 

Systems  and  Services 

SV-5c 

Operational  Activity  to  Services  Traceability  Matrix 

Systems  and  Services 

SV-6 

Systems/Services  Data  Exchange  Matrix 

Systems  and  Services 

SV-7 

Systems/Services  Performance  Parameters  Matrix 

Systems  and  Services 

SV-8 

Systems/Services  Evolution  Description 

Systems  and  Services 

SV-9 

Systems/Services  Technology  Forecast 

Systems  and  Services 

SV-lOa 

Systems/Services  Rules  Model 

Systems  and  Services 

SV-1  Ob 

Systems/Services  State  Transition  Description 

Systems  and  Services 

SV-lOc 

Systems/Services  Event-Trace  Description 

Systems  and  Services 

SV-11 

Physical  Schema 

Technical  Standards 

TV-1 

Technical  Standards  Profile 

Technical  Standards 

TV-2 

Technical  Standards  Forecast 

2.2.2  Architecture  Evaluation 

Architecture  evaluation  has  taken  many  forms  from  quantitative  scoring  measures  to 
simple  heuristics.  It  is  of  great  value  to  not  only  the  model  builder,  but  to  the  project  sponsor  as 
well,  to  be  able  to  determine  the  quality  of  a  set  of  architectural  products  and  the  associated 
instantiated  system.  Since  architectures  are  intended  to  represent  a  system,  it  is  important  to  rate 
not  only  the  architectural  products  themselves,  but  also  how  they  accomplish  the  goal  of 
representing  the  needs  of  the  system  itself. 
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One  such  evaluation  method  is  Ford’s  i-Score  (Ford,  Graham,  Colombi,  &  Jacques, 

2008),  which  measures  the  interoperability  of  the  architecture.  To  do  so,  it  uses  the  DoDAF  OV- 
5,  OV-2,  and  SV-3.  It  is  similar  to  Value-Focused  Thinking  (VFT)  in  that  it  yields  a  single 
quantitative  score  that  represents  how  the  architecture  is  performing  in  terms  of  interoperability 
(Ford,  Graham,  Colombi,  &  Jacques,  2008).  Levis,  Shin,  and  Bienvenu  (2000)  discusses  the 
concept  of  executable  architectures,  which  relies  on  modeling  and  simulation  to  detennine  the 
effectiveness  of  a  system.  There  have  been  other  tools  as  well,  such  as  the  Architecture  Based 
Evaluation  Process  (Dietrichs,  Griffin,  Schuettke,  &  Slocum,  2006)  and  the  Architecture 
Tradeoff  Analysis  MethodSM  (Kazman,  Klein,  &  Clements,  2000),  which  is  intended  to  evaluate 
software  architectures.  Of  the  architecture  evaluation  methods  in  existence,  none  attempt  to 
grade  an  architecture  based  on  “-illities,”  nor  do  they  provide  a  comprehensive  generalized 
approach  in  line  with  the  stakeholder’s  values. 

2.2.3  “Ilities” 

“Ilities”  have  grown  in  popularity  across  the  quality  management  field.  Particularly  in 
the  areas  of  software  development  and  systems  engineering,  they  have  become  standards  for 
describing  system  attributes.  The  INCOSE  Systems  Engineering  Handbook  version  3.0  defines 
ilities  as  “the  operational  and  support  requirements  a  program  must  address  (e.g.  availability, 
maintainability,  vulnerability,  reliability,  supportability,  etc.)”  (International  Council  on  Systems 
Engineering,  2007).  Within  the  systems  engineering  field,  ilities  tend  to  describe  the  quality 
attributes  of  a  system  through  their  descriptive  nature.  This  makes  ilities  useful  for  describing 
the  quality  of  both  the  instantiated  system  as  well  as  the  architectural  products.  Although  a 
single  authoritative  source  for  a  list  of  ilities  does  not  exist,  they  are  often  created  based  on  the 
quality  needs  of  the  system.  Several  studies  and  articles  refer  to  individual  use  of  ilities.  Ross 
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(2006)  discusses  ilities  (flexibility,  adaptability,  scalability,  robustness)  which  describe  the 
“traditional  system  design  concerns.”  McManus  et  al.  (2007)  created  a  quantitative  measure  for 
describing  certain  system  ilities.  Their  work  defines  and  describes  six  ilities  (robustness, 
versatility,  changeability,  flexibility,  scalability,  and  survivability)  which  share  some  of  the  same 
attributes  as  other  studies.  These  studies  begin  to  provide  a  framework  for  the  evaluation  and 
quantification  of  ilities,  although  much  more  work  is  required  in  this  area.  It  is  possible  to 
“create”  ilities  by  simply  adjusting  the  tense  of  a  system  attribute.  A  web  search  yields  one  list 
of  63  ilities,  with  many  others  scattered  throughout  various  sources. 

2.3  Decision  Analysis 

Evaluating  the  protection  status  at  a  United  States  military  installation  is  currently  a  very 
subjective  process.  Each  service  has  inspection  methods  in  place  to  ensure  compliance  with 
regulations  and  security  procedures,  but  there  is  no  quantitative,  objective  method  for  achieving 
this  goal.  Inspection  procedures  consist  of  checklists  that  are  accomplished  by  subject  matter 
experts  appointed  by  higher  headquarters,  but  their  evaluations  are  still  based  on  their  own 
subjective  understanding  of  the  regulations.  In  addition,  these  evaluations  may  not  match 
fundamental  joint  force  protection  values. 

The  Decision  Analysis  (DA)  field  provides  decision-makers  a  set  of  tools  for  making 
decisions  that  are  more  objective.  In  this  case,  DA  provides  a  quantitative,  more  objective 
approach  to  accomplishing  the  goal  of  force  protection  evaluations.  The  methods  provided 
within  the  decision  analysis  discipline  give  the  decision-maker  more  insight  to  the  problem  and 
ensure  all  data  is  being  examined,  thereby  facilitating  better  decisions.  DA  is  particularly  useful 
when  several  objectives  exist  and  affect  different  groups  of  stakeholders.  In  the  case  of 
architecture,  a  set  of  products  exist,  which  serve  to  document  a  collection  of  design  decisions. 
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Decision  analysis  provides  a  tool  to  evaluate  the  previously  made  design  decisions  as  they  relate 
to  stakeholder  values,  as  well  as  a  method  to  evaluate  decision  opportunities. 

Decision  analysis  provides  a  systematic,  iterative  approach  (shown  in  Figure  2.2)  to 
solving  a  problem  (Clemen  &  Reilly,  2001).  It  begins  with  the  most  basic  step  of  any  analysis, 
to  identify  the  decision  situation  and  understand  its  objectives.  In  this  case,  the  design  of  a  force 
protection  system  is  being  evaluated.  The  system  being  designed  will  be  subjected  to  the 
evaluations  of  the  DoD  acquisition  process.  The  evaluations  for  system  acquisition  are  largely 
subjective  and  depend  on  the  opinions  of  the  sponsor  and  acquisition  officer. 

2.4  Value-Focused  Thinking 

Value-Focused  Thinking  (VFT)  is  a  decision  making  tool  developed  by  Keeney  (1992);  it 
enables  decision-makers  to  look  beyond  the  list  of  available  alternatives  and  focus  on  the  values 
or  objectives  that  are  actually  important  to  them  in  the  outcome  of  the  situation.  The  VFT 
method  is  intended  to  get  decision-makers  closer  to  solutions  that  they  actually  want  (Keeney, 
1992).  In  a  joint  force  protection  system  situation,  this  facilitates  the  system  sponsors  getting  the 
product  they  need  as  opposed  to  choosing  between  presented  alternative  systems. 
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Figure  2.2.  Decision-Analysis  Process  Flowchart  (Clemen  &  Reilly,  2001,  p.  6) 


2.4.1  Alternative-Focused  Thinking  versus  Value-Focused  Thinking 

The  second  step  of  the  process  shown  in  Figure  2.2,  “Identify  Alternatives,”  is  the  issue 
that  defines  any  decision-making  problem.  A  decision  problem  generally  occurs  when  a 
decision-maker  is  presented  with  at  least  two  alternatives  (Keeney,  1992).  They  must  decide 
among  the  best  of  the  presented  alternatives.  This  approach  has  been  called  “Alternative- 
Focused  Thinking”  (AFT)  since  the  decision  approach  is  based  on  choosing  from  a  finite  set  of 
alternatives.  Because  AFT  happens  after  a  decision  problem  has  been  framed  and  the  solution  is 
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chosen  from  a  finite  set  of  alternatives,  it  is  a  reactive  procedure  (Keeney,  1994).  VFT,  however, 
is  a  proactive  approach  in  which  decision-makers  examine  a  problem  before  the  decision  is 
forced  upon  them.  It  enables  the  decision-makers  to  determine  what  is  important  to  them  in 
advance  of  making  the  decision  and  generate  alternatives  rather  than  choosing  from  existing 
alternatives.  This  approach  also  ensures  that  all  possible  alternatives  are  considered  instead  of 
only  a  limited  set. 

In  Alternative-Focused  Thinking  methodologies,  the  decision-maker  begins  the  process 
with  a  set  of  existing  alternatives.  Alternatives  though,  are  only  the  means  to  achieve  the  values 
of  the  decision-maker  in  this  method.  For  this  reason,  the  values  should  be  determined  before 
alternatives  are  created  (Keeney,  1994).  The  Alternative-Focused  Thinking  approach  leads  to 
less  understanding  of  what  is  important  in  the  end  goal. 

VFT  is  intended  to  lead  decision-makers  to  better  decisions.  The  process  set  forth  in 
previous  studies  and  literature  allows  for  many  other  advantages  as  well.  Among  these  are 
uncovering  hidden  objectives,  guiding  infonnation  collection,  improving  collection,  facilitating 
involvement  in  multiple-stakeholder  decision,  avoiding  conflicting  decisions,  evaluating 
alternatives,  creating  alternatives,  and  identifying  decision  opportunities  (Keeney,  1992) 

When  VFT  is  used  early  enough  in  a  process,  many  alternatives  and  opportunities  for 
improvement  are  presented  to  the  decision-maker.  In  many  cases,  a  decision  situation  is  forced 
and  does  not  leave  decision-makers  time  to  evaluate  values  and  create  an  exhaustive  list  of 
alternatives.  Instead,  they  are  presented  a  finite  list  of  alternatives  and  must  choose  the  best  of 
those  available.  By  applying  VFT  early,  it  is  possible  to  guide  infonnation  collection  and 
identify  decision  opportunities  prior  to  the  decision  situation.  During  the  design  phase  of  an 
acquisition  project,  there  are  several  opportunities  for  improvement  over  the  long  process. 
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Focusing  on  values  throughout  the  entire  process  will  allow  the  designer  to  achieve  a  more 
robust  and  useful  product  for  the  sponsor,  instead  of  simply  fulfdling  the  requirements. 

Kirkwood  (1997)  discusses  several  of  the  same  advantages,  including  that  the  VFT  process  is 
helpful  in  facilitating  communications  between  stakeholders.  VFT  seeks  to  create  a  hierarchical 
representation  of  what  is  important  to  the  decision-maker.  This  hierarchy  includes  tiers  of  value, 
which  become  more  specific  as  the  tiers  progress.  The  value  hierarchy  gives  stakeholders  a 
common  frame  of  reference  for  what  is  important  in  the  project  and  gives  all  stakeholders  input 
into  the  importance  of  each  value.  In  projects  with  a  large  number  of  objects,  complex  issues, 
and  many  stakeholders,  communication  is  essential  to  achieving  the  objectives  of  the  project. 
2.4.2  Discussion  of  Value 

Throughout  the  literature  regarding  Value-Focused  Thinking  and  Decision  Analysis,  the 
terms  “Value”  and  “Objective”  are  often  used  interchangeably.  “Values  are  what  we  care  about” 
(Keeney,  1992).  They  are  the  fundamental  part  of  any  decision  that  dictates  in  what  the 
decision-maker  is  truly  interested.  Keeney  (1992)  uses  two  distinct  terms,  value  and  objective, 
when  discussing  what  is  important.  The  tenn  “value”  refers  to  an  idea  that  the  decision-maker  is 
trying  to  describe.  The  tenn  “objective”  is  typically  used  to  describe  the  evaluation  measure  of 
the  value  included  in  the  hierarchy.  Kirkwood  (1997)  defines  the  connection  between  a  value 
and  an  objective  by  his  definition  of  the  term  “objective.”  He  defines  it  as  “the  preferred 
direction  of  movement  with  respect  to  an  evaluation  consideration.”  Several  previous  research 
efforts  have  used  either  term  to  describe  the  actual  elements  of  the  hierarchy.  Shoviak  (2001) 
uses  the  term  “objective”  almost  exclusively,  while  Katzer  (2002)  uses  “value”  in  the  same  way, 
in  the  same  contexts.  An  examination  of  several  other  VFT-focused  research  efforts  has  yielded 
similar  differences  in  the  use  of  the  terms.  Clemen  and  Reilly’s  (2001)  discussion  of  objectives 
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and  values  distinguishes  the  two  terms  based  on  their  relation  to  the  user.  They  refer  to  value  as 
anything  that  matters  to  the  decision-maker.  Objectives  are  defined  as  the  specific  things  that  the 
decision-maker  wants  to  achieve.  Based  on  these  definitions,  the  combination  of  the  objectives 
gives  the  decision-maker  their  overall  values.  The  objectives  are  the  specific  things  that  will 
influence  the  final  decision  and  the  values  are  characteristics  of  the  preferred  outcome. 

2.4.3  Value-Focused  Thinking  Methodology 

VFT  was  initially  laid  out  by  Keeney  (1992)  and  refined  by  Kirkwood  (1997).  Over  the 
years,  this  methodology  has  been  applied  and  adapted  for  several  purposes.  Keeney  (1992) 
discussed  three  “situation  based”  five-step  processes.  These  processes  refer  to  situations  in 
which  a  decision  problem  or  decision  opportunity  exists.  The  decision  opportunities  are  then 
broken  down  into  two  processes.  One  for  a  situation  before  strategic  objectives  have  been 
specified  and  one  for  situations  after  strategic  objectives  have  been  specified.  Each  of  these 
situations  have  a  five  step  process  for  completing  the  VFT  process.  These  processes,  in 
combination  with  Kirkwood  (1996)  can  be  extended  to  a  ten-step  process  (Chambal,  2001).  The 
ten-step  process  expands  on  Keeney’s  by  adding  individual  steps  for  measures,  value  functions, 
and  hierarchy  as  well  as  expanding  the  analysis  of  the  VFT  process  by  including  Kirkwood’s 
methods.  These  methods  are  shown  in  Table  2.3. 

As  demonstrated  by  the  selected  methodologies  in  Table  2.3,  there  are  several  different 
ways  to  apply  Value -Focused  Thinking  to  a  decision.  Each  of  the  steps  laid  out  in  these 
processes  share  some  similar  features,  which  can  be  combined  into  a  single,  ten-step  process, 
accounting  for  all  major  activities  and  milestones.  The  ten-step  version  guides  the  evaluator 
through  the  Keeney  and  Kirkwood  methodologies  in  a  straightforward  fashion,  ensuring  that 
each  iterative  step  accomplishes  the  necessary  activities.  The  ten-step  process  effectively 
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combines  the  previous  methodologies  and  accounts  for  all  of  their  ideas.  This  process  has  been 
refined  and  applied  in  several  research  projects.  These  projects  include  an  examination  of 
advanced  academic  degree  profiles  (Gentil,  2007),  a  Force  Protection  Battlelab  project 
evaluation  initiative  (Jurk,  2002),  and  strategic  airlift  (Tharaldson,  2006).  Chambal’s  (2008) 
process  follows  a  path  of  distinct  activities,  separating  Value  Hierarchy  creation.  Measure 
Creation,  Weighting,  Scoring,  and  Analysis  in  a  unique  way.  Figure  2.3  shows  a  graphical 
representation  of  the  ten-step  process.  In  the  following  pages,  the  VFT  process  will  be  discussed 
in  depth. 


Table  2.3.  VFT] 

Methodologies  (Keeney,  1992;  Kirkwood,  1997;  Chambal,  2001) 

Author 

Keeney 

Keeney 

Keeney 

Kirkwood 

10-Step  Process 

Situation 

Decision  Problems 

Decision  Opportunities  before 
specifying  objectives 

Decision  Opportunities  after 
specifying  objectives 

All 

All 

Steps 

Recognize  a  decision  problem 

Indentify  a  decision  opportunity 

Specify  Values 

Identify  Decision 

Problem  Identification 

Specify  Values 

Specify  Values 

Create  a  Decision  Opportunity 

Structure  Objectives 

Create  Value  Hierarchy 

Create  Alternatives 

Create  Alternatives 

Create  Alternatives 

Develop  Evaluation  Measures 

Develop  Evaluation  Measures 

Evaluate  Alternatives 

Evaluate  Alternatives 

Evaluate  Alternatives 

Develop  Alternatives 

Create  Value  Functions 

Select  an  Alternative 

Select  an  Alternative 

Select  an  Alternative 

Determine  Single  Dimensional  Value 
Function 

Weight  Hierarchy 

Develop  Weights 

Alternative  Generation 

Determine  Overall  Values  for 

Alternatives 

Alternative  Scoring 

Select  Alternative 

Deterministic  Analysis 

Sensitivity  Analysis 

Recommendations  Presentation 
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Figure  2.3.  Ten-Step  Process  Graphical  Representation  (Chambal,  2001) 


Parnell  describes  three  “levels”  of  VFT  models  as  the  Silver,  Gold,  and  Platinum 
standards  (Parnell,  2007).  These  standards  are  used  throughout  the  process  as  methods  of 
communicating  and  building  the  model.  The  framework  decided  upon  is  descriptive  of  how  the 
process  is  completed.  Several  aspects  of  the  VFT  process  are  impacted,  such  as  how  the 
hierarchy  is  built,  description  of  the  problem,  construction  of  measures,  and  perhaps  of  most 
importance,  the  development  of  value  weights.  The  Silver  standard  is  the  least  preferable  of  the 
standards  (Dawley,  Marentette,  &  Long,  2008).  It  incorporates  the  opinions  of  a  large  number  of 
stakeholders  and  uses  various  idea-generation  techniques  to  detennine  inductively  the  values  of 
the  organization  (Chambal,  2001).  The  Gold  standard  bases  the  model  construction  on  existing 
documents  and  guidance.  Through  an  examination  of  documents,  such  as  vision  statements, 
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mission  statements,  rules,  regulations,  etc.,  the  model  builder  is  able  to  gain  an  appreciation  for 
what  is  important  to  the  organization,  which  allows  deductive  development  of  the  model.  The 
organization’s  senior  leaders  must  then  validate  gold  standard  models  (Chambal,  2001). 

Through  examinations  of  documentation  as  well  as  validation,  this  standard  is  most  easily 
defendable  (Jurk,  2002).  The  Gold  standard  also  allows  the  model  builder  to  create  a 
“strawman”  hierarchy  from  which  to  begin  and  base  discussion.  Strawman  hierarchies  tend  to 
facilitate  discussions  with  the  decision-maker  and  make  effective  and  efficient  use  of  time 
(Katzer,  2002).  Finally,  the  Platinum  standard  relies  on  interviews  regarding  the  values  of  the 
key  decision-maker  as  well  as  technical  experts  and  stakeholders.  This  method  gives  not  only 
the  Subject  Matter  Experts  (SMEs)  buy-in  to  the  hierarchy,  but  also  facilitates  direct  involvement 
of  the  final  decision  authority.  The  platinum  standard  often  begins  with  an  examination  of 
strategic  objectives,  organizational  plans,  and  visions  (Chambal,  2001),  but  moves  on  to  capture 
the  values  of  the  final  decision-maker.  This  final  decision  authority’s  opinions  and  views  on  the 
system  lead  to  a  more  accurate  depiction  of  what  is  important  in  the  model.  These  models  tend 
to  capture  most  accurately  the  intended  hierarchy  structure  due  to  the  direct  involvement  of  the 
stakeholders  and  final  decision  authority  (Braziel,  2004). 

2.4.3. 1  Step  1  -  Problem  Identification 

Problem  identification  is  the  cornerstone  of  any  scientific  process  and  is  consistently 
referenced  as  the  first  step  to  solving  problems.  In  many  cases,  an  undesirable  solution  to  a 
problem  is  based  on  a  decision-maker’s  failure  to  identify  and  understand  the  problem  itself.  In 
addition  to  the  model  builders  and  decision-maker  understanding  the  objectives  of  the  process, 
all  stakeholders  should  have  a  clear  understanding  of  the  goal.  Everyone  involved  in  the  process 
must  have  a  common  idea  of  the  problem  itself,  so  that  wasted  effort  can  be  avoided. 
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2.43.2  Step  2  -  Create  Value  Hierarchy 

A  value  hierarchy  is  a  graphical  representation  of  the  values  or  objectives  most  essential 
to  the  decision-maker.  Keeney  (1994)  defines  values  as  “Principles  for  evaluating  the 
desirability  of  any  possible  alternatives  or  consequences.”  The  hierarchical  structure  allows  the 
model  builder  to  represent  values  from  a  top-down  perspective,  showing  not  only  what  the 
overarching  value  is,  but  also  going  into  the  level  of  detail  required  for  the  problem.  The 
resulting  hierarchy  must  be  defendable.  A  defendable  architecture  must  agree  with  the  decision¬ 
maker’s  objectives  as  well  as  the  organizational  goals.  This  must  also  be  done  within  the 
constraints  of  the  methodology  chosen. 

2.4.3.2.1  -  Generating  Values 

The  generation  of  values  depends  greatly  on  the  standard  chosen  for  the  model  being 
constructed.  The  actual  process  of  finding  these  values  can  be  quite  different  depending  on  the 
choice  between  the  “silver,”  “gold,”  or  “platinum”  standard.  The  values  may  come  directly  from 
an  examination  of  documentation  or  from  interviews  with  various  personnel  and  ultimately  a 
validation  by  some  level  of  decision-making  authority.  If  possible,  the  highest-level  decision¬ 
maker  should  be  chosen  (Keeney,  1994). 

Keeney  (1994)  distinguishes  between  two  different  types  of  objectives  or  values.  He 
refers  to  fundamental  objectives  as  “[objectives  that]  concern  the  ends  that  decision-makers 
value  in  a  specific  decision  context”  and  means  objectives  as  “methods  to  achieve  ends.”  Means 
objectives  serve  as  way  to  identify  fundamental  objectives.  The  means  objectives  can  be 
quantified  by  continually  asking  the  question  “Why  is  that  important?”  until  a  fundamental 
objective  is  reached  (Keeney,  1994).  Fundamental  objectives  can  also  be  called  “ends 
objectives”  (Kirkwood,  1997). 
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Developing  the  list  of  values  can  be  accomplished  through  a  number  of  techniques. 

These  techniques  include:  develop  a  wish  list;  identify  alternatives;  consider  problems  and 
shortcomings;  predict  consequences;  identify  goals,  constraints,  and  guidelines;  consider 
different  perspectives;  determine  strategic  objectives;  determine  generic  objectives;  structure 
objectives;  and  quantify  objectives  (Keeney,  1994).  Using  Keeney’s  (1994)  suggested 
techniques,  one  will  develop  a  list  of  items  including  fundamental  objectives  and  means 
objectives.  This  list  must  then  be  examined  to  determine  what  each  of  the  list  items  are,  thereby 
eliminating  items  that  are  not  values.  The  goal  of  this  process  is  to  end  with  a  list  of 
“collectively  exhaustive  and  mutually  exclusive”  objectives  (Kirkwood,  1997).  These  objectives 
should  be  characterized  by  three  features:  a  decision  context,  an  object,  and  a  direction  of 
preference. 

2.43.2.2  -  Structuring  Values 

Once  values  have  been  detennined,  they  must  be  placed  into  a  readable,  understandable, 
graphical  structure.  This  structure  allows  for  easy  communication  to  a  wider  range  of  users. 
Keeney  (1992)  notes  that  prior  to  his  work,  there  had  not  been  a  standard  fonnat  for  structuring 
values.  He  therefore  proposes  the  hierarchical  method  of  structuring  (Keeney,  1992).  Kirkwood 
(1997)  defines  a  value  hierarchy  as  “a  value  structure  with  a  hierarchical  or  “treelike”  structure.” 
This  process  is  also  known  as  a  “top-down”  structure,  as  it  is  based  on  the  fundamental  value. 

The  basic  nature  of  hierarchies  is  both  vertical  and  horizontal.  As  demonstrated  in  Figure 
2.4,  the  hierarchy  is  made  up  of  both  tiers  and  branches.  Tiers  are  the  layers  or  levels  that, 
collectively,  specify  the  values  on  the  tier  above.  Branches  are  the  values  that  actually  specify 
the  value  above.  Within  each  branch,  the  values  in  each  progressively  lower  tier  specify  a  single 
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value  from  the  above  tier  of  the  same  branch.  The  breadth  of  the  hierarchy  is  defined  by  the 
number  of  branches  and  the  depth  is  defined  by  the  number  of  tiers. 


1st  Branch 


3rd  Branch 


Figure  2.4.  Example  Hierarchy  (Chambal,  2001) 


Keeney’s  (1992)  method  involves  three  basic  steps:  placing  the  overall  fundamental 
value  at  the  top  of  the  hierarchy,  relating  values  on  different  levels,  and  stopping  the  structuring 
process.  In  this  process,  the  overall  value  is  the  reason  for  the  decision  and  defines  the  breadth 
of  the  decision  problem.  Choosing  this  overall  value  is  therefore  very  important.  For  some 
decisions,  it  is  easy  to  identify,  but  ensuring  that  it  is  the  correct  value  will  affect  the  entire 
process.  Following  the  selection  of  a  fundamental  value,  other  values  must  be  placed  below  it  in 
the  proper  branch  and  tier  based  on  their  relation  to  the  fundamental  value  (Keeney,  1992; 
Kirkwood,  1997).  Each  progressively  lower  tier  specifies  the  values  above  it.  In  Tier  2  of 
Figure  2.5,  Values  1  and  2  further  define  the  value  found  in  Value  1  of  Tier  1.  The  combination 


33 


of  the  two  lower  values  makes  up  the  higher  value.  The  critical  part  of  Keeney’s  (1992)  process 
is  stopping  the  structuring  process.  The  “test  of  importance”  is  the  first  detennination  as  to  how 
many  values  should  be  included.  The  size  of  the  hierarchy  must  be  balanced  with  the  detail 
necessary  to  capture  the  values.  Each  value  must  also  be  measurable  by  an  attribute  (Keeney, 
1992).  The  model  builder  should  continue  moving  down  the  hierarchy,  progressively  refining 
the  values  within  each  branch  by  adding  more  tiers,  until  the  model  builder  no  longer  must  ask 
“What  do  you  mean  by  that?”  (Katzer,  2002).  Moving  up  the  hierarchy  within  a  branch  answers 
the  question,  “of  what  more  general  objective  is  this  an  aspect?”  (Katzer,  2002).  When  building 
the  value  hierarchy,  if  a  value  cannot  be  decomposed  into  more  than  one  lower  tier  value,  it 
should  not  be  decomposed.  As  the  number  of  tiers  within  a  hierarchy  increases,  its  size  increases 
vertically. 

2.4.3.2.3  -  Desirable  Properties  of  a  Value  Hierarchy 

Kirkwood  (1997)  presents  five  properties  that  are  desirable  for  any  value  hierarchy. 

These  properties  include  completeness,  nonredundancy,  decomposability,  operability,  and  small 
size.  Completeness  is  considered  one  of  the  most  important  properties  for  a  hierarchy  to  exhibit. 
Another  way  of  describing  completeness  is  if  a  hierarchy  is  “collectively  exhaustive.”  It  stands 
to  reason  that  any  hierarchy  must  contain  all  of  the  values  important  to  the  decision-maker.  This 
includes  any  value  that  is  required  to  evaluate  the  fundamental  objective.  For  a  hierarchy  to 
show  completeness,  it  must  be  possible  to  evaluate  the  objective  based  only  on  the  values 
presented  in  the  hierarchy.  If  there  are  other  considerations  required  for  evaluation,  they  must  be 
added  to  the  hierarchy.  This  includes  all  values,  no  matter  how  small  of  a  part  they  may  play  in 
the  final  evaluation.  Their  magnitude  of  importance  is  considered  during  the  weighting  phase  of 
the  evaluation  (Kirkwood,  1997). 
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There  may  also  exist  values  that  appear  to  be  more  basic  to  the  problem  than  evaluation 
criterion.  These  values  may  be  “promoted”  to  “screening  criteria.”  Screening  criteria  will  be 
discussed  in  more  depth  in  Section  2. 4. 3. 6  -  Alternative  Generation.  Determining  the  difference 
between  values  and  screening  criteria  may  be  difficult.  Screening  criteria  should  be  used  for  the 
sole  purpose  of  reducing  the  number  of  alternatives  (Kirkwood,  1997).  For  example,  a  value 
may  be  “Distance”  while  a  screening  criteria  would  be  “Less  than  five  miles  away.”  The 
distance  value  simply  states  that  the  relative  distance  is  important  in  the  end  decision.  The 
screening  criteria  “Less  than  live  miles  away”  specifically  eliminates  certain  alternatives  based 
on  their  distance  (Kirkwood,  1997).  Each  value  must  also  pass  a  “test  of  importance.”  The 
decision-maker  must  ask  whether  the  inclusion  of  a  specific  value  will  alter  the  outcome  of  the 
decision  problem.  If  the  decision-maker  feels  that  the  exclusion  of  a  specific  value  could  alter 
the  best  course  of  action,  then  it  must  be  included.  If  the  exclusion  of  a  value  will  have  no  effect 
on  the  outcome,  then  it  can  be  left  out  of  the  hierarchy.  The  major  caution  with  this  “test  of 
importance”  is  the  possible  exclusion  of  a  collection  of  independently  unimportant  values,  but 
which  serve  a  major  part  in  the  decision  when  considered  together.  The  collective  importance  of 
any  excluded  values  must  be  continually  evaluated;  therefore,  the  excluded  values  should  not  be 
completely  discarded,  so  that  future  iterative  evaluations  may  be  completed  on  them.  By 
conducting  the  test  of  importance  at  different  stages  in  the  process  and  with  the  obvious 
groupings  of  these  values,  it  is  possible  to  ensure  that  none  of  the  excluded  values  will  have  a 
major  effect  on  the  final  decision  (Keeney  &  Raiffa,  1993). 

Nonredundancy  is  another  very  important  property  of  a  value  hierarchy.  Nonredundancy 
is  also  referred  to  as  “mutually  exclusivity”  (Kirkwood,  1997).  A  hierarchy  is  considered 
mutually  exclusive  if  no  two  values  in  the  same  tier  overlap  in  any  way.  Every  aspect  of  the 
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evaluation  criteria  must  relate  to  one  and  only  one  value.  Each  tier  of  the  hierarchy  should 
divide  the  tier  above,  lower  levels  essentially  composing  the  values  above  them.  The  property  of 
nonredundancy  ensures  that  no  value  is  double-counted  and  therefore  receives  more  weight  than 
it  deserves.  Based  on  these  first  two  properties,  every  hierarchy  at  its  very  base  must  be 
“collectively  exhaustive  and  mutually  exclusive.”  These  are  possibly  the  two  most  important 
properties,  since  they  ensure  that  everything  is  included  and  that  all  values  required  are 
represented  only  once  in  the  hierarchy. 

Decomposability  or  independence  refers  to  a  value’s  influence  on  other  values 
(Kirkwood,  1997).  The  actual  scoring  of  any  value  cannot  have  any  influence  on  the  scoring  of 
another  value.  Decomposability  ensures  that  there  is  not  only  a  measurement  for  each  value,  but 
that  the  measurements  themselves  are  also  mutually  exclusive.  This  separation  of  measurement 
ensures  that  the  weighting  of  each  value  may  be  completed  (Shoviak,  2001). 

An  operable  hierarchy  refers  more  to  the  utility  of  the  tool  itself.  Any  value  hierarchy 
must  be  understandable,  at  a  minimum  to  those  who  must  use  it  in  an  evaluation  (Kirkwood, 
1997).  This  is  a  rather  subjective  property,  but  in  communicating  a  hierarchy  or  decision 
analysis  tool,  the  users  must  be  able  to  understand  quickly  and  easily  the  points  that  the  model 
builder  is  trying  to  get  across.  The  more  technical  the  subject  matter,  the  more  difficult  it  will  be 
to  satisfy  this  property,  although  the  subject  matter  itself  does  not  necessarily  have  to  have  an 
effect  on  the  understandability  of  the  hierarchy  itself.  If  the  reader  is  able  to  understand  the  tool, 
then  it  is  considered  operable. 

The  last  desirable  characteristic  of  a  value  hierarchy  is  small  size.  In  comparison,  a 
smaller  hierarchy  is  generally  preferable  to  a  larger  one  (Kirkwood,  1997).  The  key  tradeoff  is 
that  it  must  also  be  collectively  exhaustive.  Therefore,  the  small  size  property  is  directly  dictated 
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by  the  minimum  amount  of  infonnation  necessary  to  properly  evaluate  the  decision  problem. 
Small  size  also  has  an  influence  on  the  operability  of  the  hierarchy.  Larger  hierarchies  are 
generally  more  difficult  to  communicate  to  stakeholders  than  a  compact  hierarchy.  The  size  also 
becomes  an  issue  in  the  employment  and  analysis  of  the  tool.  A  larger,  more  complex  tool  will 
be  not  only  more  difficult  to  use,  but  more  difficult  to  evaluate. 

The  size  issue  must  be  considered  in  two  dimensions.  The  nature  of  hierarchies  is  both 
horizontal  and  vertical.  Therefore,  the  model  builder  must  be  sure  to  include  not  only  the 
necessary  breadth,  but  depth  as  well.  Breadth  of  the  model  is  detennined  by  the  number  of 
values  to  which  the  fundamental  value  can  be  decomposed.  As  the  level  of  tiers  increases,  the 
number  of  values  generally  increases  exponentially.  Therefore,  an  increase  in  depth  has  a  direct 
effect  on  the  breadth  of  the  hierarchy.  To  keep  these  two  issues  under  control,  the  “test  of 
importance”  and  guidance  by  the  model  builder  must  be  used  to  ensure  that  each  tier  and  value 
are  directly  influential  to  the  fundamental  value. 

2.4.3.3  Step  3  -  Develop  Evaluation  Measures 

Evaluation  measures  or  attributes  exist  for  detennining  how  well  an  alternative  perfonns 
with  respect  to  a  particular  value.  This  can  be  accomplished  qualitatively  or  quantitatively,  but 
each  lowest-tier  value  must  be  measurable.  The  evaluation  measures  should  provide  the 
mechanism  for  turning  a  subjective  decision  into  an  objective  decision  (Kirkwood,  1997). 
Graphically,  the  measures  appear  below  the  lowest  level  of  decomposition  in  the  value  hierarchy. 
The  model  builder  should  use  as  many  measures  as  necessary  to  properly  quantify  the  attributes 
of  the  value. 


37 


2.4.3.3.1  -  Types  of  Evaluation  Measures 

Evaluation  measures  may  be  classified  as  either  natural  or  constructed  and  direct  or 
proxy.  The  type  of  measure  depends  upon  the  availability  of  data  regarding  the  value  as  well  as 
whether  the  measure  is  qualitative  or  quantitative.  A  natural  scale  is  one  with  a  common 
interpretation  to  any  audience.  Constructed  scales  are  developed  specifically  for  measuring  the 
value  (Kirkwood,  1997).  Generally,  constructed  scales  are  used  when  no  natural  scale  is  evident 
or  they  may  also  be  used  when  there  is  not  enough  data  to  measure  the  value  exactly  (Kirkwood, 
1997). 

In  addition  to  being  either  natural  or  constructed,  a  measure  will  also  be  either  direct  or 
proxy.  Direct  scales  measure  the  degree  of  attainment  of  the  value  explicitly.  A  proxy  measure 
still  measures  the  value,  but  does  so  indirectly.  Proxy  measurement  may  use  some  other  piece  of 
data  or  a  collection  of  data  that  represents  the  degree  of  attainment  of  the  value.  It  is  possible  for 
any  measure  to  be  categorized  as  any  combination  of  natural/constructed  and  direct/proxy 
(Kirkwood,  1997).  A  Natural-Direct  measure  is  the  most  preferable,  as  it  measures  the  value 
most  accurately.  Natural-Proxy  and  Constructed-Direct  are  next  in  terms  of  desirability,  and  a 
Constructed-Proxy  scale  is  the  least  desirable  since  it  requires  interpolation  between  the  measure 
and  the  value  (Table  2.4)  (Dawley,  Marentette,  &  Long,  2008). 

2.4.3.3.2  -  Desirable  Properties  of  Evaluation  Measures 

Just  as  with  values,  several  properties  are  desirable  for  evaluation  measures.  In  the  case 
of  the  measures,  model  builders  should  consider  measurability,  operationality,  and 
understandability.  Measurability  “defines  the  associated  value  in  more  detail  than  that  provided 
by  the  value  alone”  (Keeney,  1992).  Each  measure  must  define  the  value  as  intended  by  the 
decision-maker.  Operationality  refers  to  a  measure’s  ability  to  “express  relative  preferences  for 
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different  levels  of  achievement  of  an  objective”  (Keeney,  1992).  Finally,  a  measure  is 
considered  understandable  if  any  audience  can  easily  understand  its  purpose  as  was  originally 
intended  by  the  model  builder. 

2.4.3.4  Step  4  -  Value  Function  Creation 

Value  functions  exist  for  the  purpose  of  converting  the  measurement  of  an  objective  into 
value  units.  Converting  the  measurements  into  value  solves  the  problem  of  the  values  being 
measured  with  different  scales  and  different  units.  A  Single  Dimension  Value  Function  (SDVF) 
plots  the  measurement  of  the  value  (x-axis)  versus  a  related  value  unit  from  zero  to  one  (y-axis) 
(Kirkwood,  1997).  The  least  preferred  score  of  a  measurement  will  relate  to  zero,  while  the  best 
possible  score  earns  a  full  value  of  one.  An  alternative’s  degree  of  attainment  of  the  value  in 
question  will  be  plotted  on  the  x-axis  of  the  measure.  SDVFs  are  built  using  inputs  from  the 
decision-makers,  stakeholders,  or  available  data  on  the  values. 

SDVFs  are  defined  by  their  shape  and  monotonicity;  they  may  also  be  either  continuous 
or  discrete.  Continuous  SDVFs  are  either  monotonically  increasing  or  monotonically 
decreasing.  Figure  2.5  is  an  example  of  a  linear,  monotonically  increasing  SDVF,  meaning  that 
the  difference  in  value  between  10  and  20  on  the  x-axis  is  the  same  as  the  difference  between  70 
and  80.  Figures  2.6  and  2.7  are  examples  of  exponential  monotonically  increasing  SDVFs.  In 
the  case  of  Figure  2.6,  the  difference  in  value  between  10  and  20  on  the  x-axis  is  considerably 
smaller  than  the  difference  between  70  and  80.  This  is  referred  to  as  a  “convex”  exponential 
curve.  Figure  2.7  is  referred  to  as  “concave,”  and  exhibits  similar  properties  as  a  concave  SDVF. 
Therefore,  in  this  case,  as  more  of  the  score  is  attained,  the  value  gained  gets  smaller  (more  value 
is  earned  early),  whereas  in  Figure  2.6,  as  more  of  the  score  is  attained,  the  value  gets 
exponentially  larger  (Kirkwood,  1997).  Figures  2.8,  2.9,  and  2.10  are  all  examples  of 
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monotonically  decreasing  SDVFs;  measure  5  is  a  linear  decreasing  SDVF  while  measures  5  and 
6  are  exponentially  decreasing  SDVFs.  (Kirkwood,  1997).  The  decreasing  SDVFs  are 
interpreted  the  same  way  as  increasing  SDVFs. 

Figure  2.11  is  an  example  of  a  piecewise  linear  SDVF.  Piecewise  linear  curves  may  also 
be  monotonically  increasing  or  decreasing.  They  are  composed  of  multiple  linear  sections  that 
are  broken  by  inflection  points.  In  the  example  measure,  value  is  earned  more  quickly  between 
x-axis  values  of  25  to  70,  value  is  earned  more  slowly  when  the  x-axis  values  are  smaller  than  25 
or  greater  than  70  (Kirkwood,  1997). 

Figures  2.12,  2.13,  2.14,  and  2.15  are  examples  of  “S-Curve”  SDVFs.  “S-Curve”  SDVFs 
are  a  type  of  exponential  curve  which  may  also  be  either  monotonically  increasing  or  decreasing, 
but  take  on  the  properties  of  a  piecewise  curve  while  retaining  the  exponential  shape.  The  four 
example  measures  shown  are  the  four  possible  general  configurations  of  S-Curves.  They 
account  for  both  monotonically  increasing  and  decreasing  curves  as  well  as  concave  and  convex 
shapes. 
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Figure  2.5.  Monotonically  Increasing  Figure  2.6.  Monotonically  Increasing 


Figure  2.7.  Monotonically  Increasing  Figure  2.8.  Monotonically  Decreasing 


Figure  2.9.  Monotonically  Decreasing  Figure  2.10.  Monotonically  Decreasing 
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Figure  2.11-  Piecewise  Linear  Figure  2.12.  Monotonically  Increasing 


Figure  2.13.  Monotonically  Increasing  Figure  2. 14.  Monotonically  Decreasing 


Figure  2.15.  Monotonically  Decreasing 


Figure  2.16.  Discrete 
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The  final  possible  type  of  SDVF  is  discrete.  In  this  style  of  value  function,  the  possible 
scores  are  grouped  into  categories  or  bins.  The  value,  therefore,  increases  incrementally  to 
account  for  the  changes  in  categories.  This  type  of  SDVF  is  particularly  useful  for  qualitative  or 
binary  measures.  The  categories  must  be  well  defined  so  that  there  is  no  question  as  to  which 
category  an  alternative  belongs.  Figure  2.16  shows  an  example  of  a  discrete  measure. 

2.4.3.5  Step  5  -  Value  Hierarchy  Weights 

With  the  value  hierarchy  and  SDVFs  created,  the  decision-maker  has  a  solid  frame  of 
reference  for  what  is  important,  as  well  as  a  basis  for  the  values  implicit  in  the  decision.  Each 
value  must  then  be  weighted  to  show  its  relative  importance  to  the  decision-maker.  There  are 
two  primary  methods  for  detennining  the  weight  of  each  value,  the  direct  weighting  method  and 
the  swing  weighting  method.  Both  of  the  methods  give  rise  to  local  weights  and  global  weights 
(Shoviak,  2001). 

Local  weight  refers  to  the  relative  importance  of  a  single  value  in  relation  to  other  values 
in  the  same  branch  and  tier.  Therefore,  the  values  in  each  branch  and  tier  must  sum  to  one.  In 
the  case  of  Figure  2.5,  all  values  in  Tier  1  must  total  1.  Therefore,  Tier  1  Value  1  may  have  a 
weight  of  0.6,  while  Tier  1  Value  2  has  a  weight  of  0.3  and  Tier  1  Value  3  has  a  weight  of  0.1. 
This  means  that  Tier  1  Value  1  is  twice  as  important  as  Tier  1  Value  2,  and  Tier  1  Value  2  is  3 
times  as  important  as  Tier  1  Value  3.  This  method  is  applied  to  each  tier  and  branch  of  the 
hierarchy.  Therefore,  the  weight  of  Tier  2  Value  1  and  Tier  2  Value  2  must  also  sum  to  1  to 
make  up  the  total  value  of  Tier  1  Value  1.  Measures  are  also  weighted  in  the  same  way.  As  the 
process  moves,  the  model  builder  and  decision-maker  may  weight  the  hierarchy  moving  from  the 
lowest  tier  to  the  highest  or  from  the  highest  to  the  lowest. 
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One  method  for  leading  the  decision-maker  to  a  conclusion  of  the  weights  is  the  use  of 
the  “100  Coin”  method  (Jurk,  2002).  In  this  situation,  the  decision-maker  is  asked  to  distribute 
100  “coins”  between  the  values;  i.e.  if  the  decision-maker  had  100  coins  to  distribute  between 
the  different  values,  where  would  they  be  placed?  In  this  method,  the  number  of  “coins”  placed 
on  any  value  becomes  the  percentage  of  importance  or  the  percentage  of  emphasis  placed  on  one 
value  when  compared  to  others  in  the  same  tier  and  branch.  Decision-makers  may  also  be  asked 
to  rate  each  value  relative  to  the  others.  For  example,  the  decision-maker  may  say  that  “Tier  1 
Value  1  is  twice  as  important  as  Tier  1  Value  2  and  Tier  1  Value  2  is  3  times  as  important  as  Tier 
1  Value  3.”  In  this  case,  the  weights  become  0.6,  0.3,  and  0.1,  respectively.  Ratios  may  also  be 
used  to  detennine  the  weights. 

Local  weights  determine  the  relative  importance  of  values  in  relation  to  the  other  values 
on  the  same  tier,  but  values  must  also  be  rated  in  tenns  of  “Global”  importance.  These  weights 
are  referred  to  as  “Global  Weights.”  The  global  weight  may  be  found  through  direct  weighting 
or  it  may  be  found  after  local  weighting  by  their  multiplicative  functions  in  relation  to  the  overall 
fundamental  value  in  the  hierarchy.  Global  weights  must  sum  to  1  across  an  entire  tier  as 
opposed  to  local  weights,  whose  sum  must  be  one  for  a  tier  in  any  given  branch.  Consider  a  case 
in  which  Tier  2  Value  1  ’s  local  weight  is  0.75  and  Tier  2  Value  2’s  local  weight  is  0.25  locally. 

In  this  case,  the  weights  are  multiplied  up  the  hierarchy  to  determine  their  global  importance.  If 
Tier  1  Value  1  ’s  weight  is  0.6,  then  the  global  weights  for  Tier  2  Value  1  and  Tier  2  Value  2 
become  0.45  (x  =  0.75*0.6)  and  0.15  (x  =  0.25*0.6),  respectively.  Figure  2.17  shows  the 
example  hierarchy  with  local  weights  displayed  and  Figure  2.18  shows  the  hierarchy  with  global 
weights  displayed.  As  is  evident  here,  all  values  in  a  tier  total  1  and  the  measures  are  then 
weighted  according  to  the  value  that  they  measure. 
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Figure  2.17.  Example  Hierarchy  with  Local  Weights 


Fundamental 

Value 

1.000 


Tier  1  Value  1 
0.600 


Tier  1  Value  2 
0.300 

Tier  1  Value  3 
0.100 

Tier  2  Value  1 
0.450 

Tier  2  Value  2 

0.150 

Tier  2  Value  3 
0.150 

Tier  2  Value  4 

0.150 

Tier  2  Value  5 

0.050 

Tier  2  Value  6 

0.050 

Figure  2.18.  Example  Hierarchy  with  Global  Weights 
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Another  method  of  weighting  a  hierarchy  is  known  as  “Swing  Weighting.”  This  method 
is  a  local  weighting  technique  and  was  compiled  from  procedures  set  forth  by  Chambal  (2008) 
and  Kirkwood  (1997).  This  technique  examines  the  possible  outcomes  that  may  be  reached 
based  on  the  weights  of  the  values.  The  decision-maker  is  asked  to  examine  each  tier  of  values 
and  determine  the  change  in  increments  of  value  that  would  be  reached  by  varying  the  weight  of 
each  value  from  its  least  preferred  state  to  its  most  preferred  state.  These  increments  are  then 
placed  in  increasing  order  and  assigned  a  factor  of  importance  in  relation  to  the  smallest  value. 
These  increments,  which  should  sum  to  one,  are  then  solved  as  a  system  of  equations  to 
detennine  the  local  weight  within  the  given  tier  (Jurk,  2002). 

2.4.3.6  Step  6  -  Alternative  Generation 

One  of  the  advantages  of  using  VFT  is  the  ability  to  generate  alternatives  as  opposed  to 
simply  choosing  from  given  alternatives.  Once  the  hierarchy  has  been  weighted,  this  is  possible. 
In  the  initial  stages  of  alternative  generation,  experience  gained  by  simply  creating  the  hierarchy 
will  often  yield  a  great  number  of  possible  alternatives.  Building  the  hierarchy  often  gives  the 
decision-maker  new  ideas  and  insights  into  the  importance  of  the  outcome  and  new  ideas  for 
alternatives.  “Either  the  alternatives  are  somewhere  in  the  mind  waiting  to  be  found,  or  they  can 
be  created  from  what  is  in  the  mind”  (Keeney,  1992,  p.  198). 

If  too  many  alternatives  are  found,  the  list  must  be  reduced  to  a  manageable  number.  In 
this  case,  additional  screening  criteria  may  be  added  to  eliminate  some  of  the  less  desirable 
options.  Screening  criteria  are  based  on  values  that  serve  to  eliminate  some  alternatives  prior  to 
scoring.  A  screening  criteria  may  be  established  if  some  alternative  scores  zero  on  a  particular 
measure.  Screening  criteria  may  also  be  something  that  is  required  by  the  decision-maker;  i.e.,  if 
some  value  or  condition  is  not  true,  the  alternative  is  eliminated  completely  from  consideration. 
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Some  values  may  be  so  important  that  an  alternative  will  not  be  considered  without  their 
inclusion.  Alternatives  may  also  be  eliminated  based  on  known  values.  If  there  are  not  enough 
alternatives,  this  usually  suggests  a  gap  in  the  value  hierarchy,  i.e.,  there  is  something  important 
which  is  not  being  considered  and  that  would  give  the  decision-maker  more  alternatives. 

Strategy  generation  tables  may  also  be  used  to  generate  alternatives  (Kirkwood,  1997).  The  most 
important  thing  to  remember  during  this  process  is  that  the  alternatives  must  satisfy  the  values  in 
the  hierarchy.  In  some  cases,  alternative  generation  may  not  be  necessary  if  the  field  of 
alternatives  is  given  or  if  some  outside  factor  limits  the  alternatives. 

2.4.3. 7  Step  7  -  Alternative  Scoring 

Following  alternative  generation,  each  alternative  must  be  individually  scored.  Data  is 
collected  regarding  each  alternative  and  its  attainment  of  each  lowest-tier  value  (based  on  the 
measures  of  those  values).  Scores  are  then  assigned  to  each  measure  within  each  alternative. 
During  this  process,  the  y-axis  or  value  units  are  hidden  from  the  scorers,  so  that  the  value  does 
not  impact  the  scoring  of  the  alternatives.  Each  score  must  be  well  documented,  clearly  defined, 
and  repeatable  by  anyone  who  scores  the  alternative. 

2.4.3. 8  Step  8  -  Deterministic  Analysis 

The  deterministic  analysis  step  combines  all  data  collected  to  this  point.  Through  the  use 
of  an  additive  value  function,  the  scores  given  to  each  alternative  (step  7)  are  converted  to  value 
units  (step  4),  and  then  multiplied  by  their  weights  (step  5)  to  yield  a  single  aggregate  score.  The 
additive  value  function  is  the  way  in  which  the  decision-maker  may  perform  detailed  analysis  of 
the  alternatives  (Shoviak,  2001).  The  general  additive  value  is  described  in  equation  2. 1 
(Kirkwood,  1997): 
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v(x)  =  Id=1Aivi(.xi) 


(2.1) 


Where  v(x)  is  the  overall  score  of  the  alternative,  ry  (Xj)  is  the  value  of  the  score  on  the  ith 
measure,  is  weight  of  the  ith  measure,  n  is  the  total  number  of  measure,  and  the  sum  or  all  At 
must  equal  one. 

2.4.3.9  Step  9  -  Sensitivity  Analysis 

Sensitivity  analysis  is  perfonned  on  the  hierarchy  to  provide  additional  insight  into  the 
weighting  of  the  values  and  how  they  affect  the  scores  of  the  alternatives.  Typically,  sensitivity 
analysis  is  perfonned  on  the  higher  tiers  of  the  hierarchy,  since  altering  the  weights  of  values  on 
lower  tiers  will  have  less  effect  on  the  total  score.  Sensitivity  analysis  is  performed  by 
systematically  altering  the  weight  (local  or  global)  of  one  value,  while  keeping  the  other  weights 
on  that  tier  proportional.  The  weights  must  continue  to  sum  to  one  across  a  tier.  Sensitivity 
analysis  serves  to  answer  the  question,  “How  would  this  decision  change  if  another  interested 
party  had  weighted  the  hierarchy  or  provided  data  for  the  SDVFs?”  (Katzer,  2002,  p.  46). 

2.4.3.10  Step  10  -  Recommendations  Presentation 

The  final  step  in  the  process  requires  the  model  builder  to  present  recommendations  to 
the  decision-maker.  Parnell  suggests  that  one -third  of  decision  analysis  efforts  should  be  placed 
in  the  recommendations  presentation.  The  recommendations  must  be  easy  to  understand  for  all 
audiences.  They  must  also  explain  the  decision  made  and  why  it  was  made.  It  is  important  to 
remember  that  the  final  decision  still  lies  in  the  hands  of  the  final  decision  authority.  The  VFT 
process  serves  to  assist  the  decision-making  process  and  provide  objective  data  and  an  analysis 
of  alternatives.  There  may  be  cases  where  the  recommended  alternative  may  not  be  chosen. 
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2.5.  Management  and  Planning  Tools 

As  a  part  of  the  value  determination  process,  it  may  be  necessary  to  organize  infonnation 
found  during  the  document  review  phase.  Several  tools  exist  for  managing  ideas  and  concepts. 
Tague  (2004)  identified  seven  management  and  planning  tools;  these  consist  of  affinity 
diagrams,  interrelationship  diagrams,  tree  diagrams,  prioritization  matrices,  matrix  diagrams, 
process  decision  program  charts,  and  activity  network  diagrams.  These  tools  were  organized  in 
1976  in  an  effort  to  collect  quality  control  techniques  (Tague,  2004).  All  of  them  were  not 
created  at  this  point,  but  were  put  together  in  a  single  work  for  managers  to  easily  locate.  The 
tools  allow  managers  to  organize  ideas  and  concepts  to  make  better,  more  efficient  decisions, 
which  take  into  account  all  known  information. 

2.5.1  Affinity  Diagrams 

The  affinity  diagram,  developed  by  Jiro  Kawakita  in  the  1960s,  was  created  to  expound 
on  the  brainstonning  group  creativity  technique.  In  brainstorming,  groups  of  people  come 
together  and  generate  as  many  ideas  as  possible  related  to  a  single  concept.  This  method  focuses 
on  the  power  of  the  group  to  generate  a  larger  quantity  of  ideas  than  any  individual  can  (Osborn, 
1953).  Affinity  diagramming  takes  this  more  generalized  approach  and  improves  upon  it.  The 
brainstonning  process  is  used  to  initially  generate  ideas,  but  through  a  process  of  organization 
and  idea  mapping,  combined  with  subsequent  discussion,  the  ideas  are  eventually  sorted  into 
descriptive  groups.  Affinity  diagrams  are  used  in  situations  when  there  are  a  great  many  ideas  or 
issues,  which  are  in  no  apparent  order  and  complex  in  nature  (Tague,  2004). 

The  decision-maker  benefits  from  the  large  quantity  of  ideas  generated  by  the  team  by 
using  the  affinity  diagramming  technique.  The  process  begins  by  describing  the  problem  and 
ensuring  that  all  team  members  are  familiar  with  the  issues.  The  brainstonning  technique  is  then 
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used  to  gather  as  many  ideas  as  possible.  These  ideas  are  then  written  on  individual  note  cards 
or  “sticky  notes.”  The  members  of  the  affinity  diagramming  team  then  physically  begin  to 
silently  organize  the  notes  into  groups.  Each  person  in  the  process  may  have  their  own  ideas,  so 
some  notes  are  moved  several  times,  but  no  discussion  is  allowed  during  this  process.  Once  all 
notes  are  placed  in  groups  or  set  aside  for  discussion,  the  resulting  groups  are  discussed  and 
examined  by  the  team.  Finally,  any  “super  groups”  that  have  emerged  are  created  by  defining 
the  individual  groups  and  further  organizing  the  cards.  The  end  result  of  this  process  is  a  number 
of  groups  and  possibly  hierarchies  which  describe  ideas  (Tague,  2004). 

The  Value-Focused  Thinking  process  involves  a  top-down  analysis  approach,  but  at 
times,  it  is  difficult  to  determine  the  lowest-level  tier  values.  Affinity  diagramming  provides  a 
method  for  combining  a  bottom-up  approach  to  the  existing  process  to  ensure  accurate  definition 
of  the  lowest  tier  (Pruitt,  2003).  Affinity  diagramming  is  an  appropriate  technique,  due  to  its 
ability  to  organize  large  amounts  of  complex  information  into  groups  with  sub-categories  being 
built  into  the  technique. 

2.5.2  Other  tools 

In  addition  to  affinity  diagramming,  there  are  several  other  tools  commonly  used  in  the 
management  and  planning  industry  (Tague,  2004).  Each  of  these  tools  is  used  for  a  very  specific 
purpose.  Interrelationship  diagrams,  for  example,  describe  the  links  and  interfaces  between 
ideas.  They  serve  to  identify  any  cause  and  effect  relationships  that  exist.  They  are  also  used  for 
complex  issues,  but  are  generally  used  as  a  follow-on  to  affinity  diagramming  when  cause  and 
effect  relationships  must  be  defined  (Tague,  2004).  Tree  diagrams  may  also  be  used  to 
breakdown  more  general  ideas  into  their  components.  Tree  diagrams  often  depend  on  affinity 
diagrams  to  first  identify  the  issues  upon  which  to  expand  (Tague,  2004).  Matrix  diagrams  are 
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another  way  of  showing  relationships  between  ideas,  but  they  organize  the  relationships 
differently  than  an  interrelationship  diagram.  Matrix  diagrams  can  also  show  relationships 
between  multiple  groups  of  information,  while  including  specific  information  regarding  the 
relationship.  They  can  be  categorized  into  six  different  “shapes,”  which  can  be  used  for  different 
numbers  of  groups  and  different  types  of  relationships  (Tague,  2004).  Matrix  data  analysis  then 
allows  the  decision-maker  to  perform  complex  mathematical  analyses  on  the  resulting  matrices 
(Tague,  2004).  “L-Shaped”  matrices  are  often  used  to  prioritize  ideas  (Tague,  2004).  An  arrow 
diagram;  also  known  as  program  evaluation  and  review  technique  (PERT)  chart,  network 
diagram,  activity  chart,  critical  path  method  (CPM),  or  node  diagram;  is  used  to  describe  the 
order  of  tasks.  Arrow  diagrams  are  very  useful  in  showing  chronological  order  of  ideas.  They 
can  describe  when  tasks  precede  others  as  well  as  durations  (Tague,  2004). 

2.6  Net-Centricity 

Net-centricity  refers  to  the  process  by  which  several  nodes  in  communication  with  each 
other  operate.  In  the  evolving  technological  world,  it  is  increasingly  important  that  the  complex 
network  of  personnel,  devices,  services,  and  information  be  connected.  The  speed  of 
communication  and  efficiency  of  passing  information  between  this  complex  system  of  nodes  has 
a  major  part  in  the  decision-making  process.  The  infonnation  age  has  given  society  access  to  a 
large  amount  of  previously  unavailable  infonnation;  it  is  now  a  matter  of  ensuring  that  the 
technology  and  infrastructure  exists  to  move  the  information  to  the  correct  person  at  the  conect 
time. 
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Net-centricity  has  given  way  to  Net-Centric  Warfare  (NCW)  within  the  U.S.  military. 

NCW  is  defined  by  Albert  et  al.  (2000)  as, 

an  infonnation  superiority-enabled  concept  of  operations  that  generates  increased 
combat  power  by  networking  sensors,  decision-makers,  and  shooters  to  achieve 
shared  awareness,  increased  speed  of  command,  higher  tempo  of  operations, 
greater  lethality,  increased  survivability,  and  a  degree  of  self-synchronization. 

The  concept  of  NCW  is  the  idea  of  linking  nodes  to  transfer  information,  thereby  ensuring 

information  superiority  for  the  warfighter  (Alberts,  Garstka,  &  Stein,  2000). 

2.6.1  Net-Centric  Enterprise  Architecture 

As  systems  architecture  grows,  the  importance  of  capturing  the  communication  between 

nodes  in  a  relevant  manner  is  becoming  increasingly  important.  To  emphasize  this  idea,  Net- 

Centric  Enterprise  Architectures  are  becoming  the  standard  in  the  systems  engineering  field.  A 

net-centric  enterprise  architecture  is  formally  defined  by  Nzuwah  (2003)  as, 

a  light-weight,  massively  distributed,  horizontally- applied  client/server 
architecture,  that  distributes  components  and/or  services  across  an  enterprise’s 
infonnation  value  chain  using  internet  technologies  and  other  network  protocols 
as  the  principal  mechanism  for  supporting  the  distribution  and  processing  of 
infonnation  services. 

The  concept  of  a  net-centric  architecture,  therefore,  is  any  architecture  that  makes  use  of 
technology  to  ensure  the  proper  communication  of  all  nodes  in  the  system.  In  the  case  of 
systems  architecture,  this  can  refer  to  not  only  the  products  themselves,  but  to  the  development 
of  the  products  and  the  net-centricity  of  the  instantiated  system.  The  system  being  represented 
by  the  architecture  must  also  hold  to  the  principles  of  net-centricity. 

2.6.2  Net-Centric  Enterprise  Solutions  for  Interoperability  (NESI) 

Currently,  the  DoD  does  not  explicitly  require  the  implementation  of  net-centricity; 
however,  it  has  made  the  intended  direction  to  move  toward  it  obvious.  The  DoD’s  Chief 
Information  Officer  (CIO)  has  stated  a  goal  to  integrate  data  into  a  central  network  and  change 
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the  paradigm  from  data  “push”  to  “pull”  (Department  of  Defense  Chief  Info nnation  Officer, 
2003).  To  this  end,  the  U.S.  Navy  Program  Executive  Office  for  Command,  Control, 
Communications,  Computers,  and  Intelligence,  in  cooperation  with  the  U.S.  Air  Force  Electronic 
Systems  Center  and  the  Defense  Information  Systems  Agency  has  produced  the  Net-Centric 
Enterprise  Solutions  for  Interoperability  (NESI)  as  a  series  of  guidance  documents.  NESI 
provides  cradle-to-grave  actionable  guidance  for  the  implementation  of  net-centric  systems  that 
meets  the  goals  set  forth  by  the  DoD  CIO.  NESI  pulls  together  several  sources  to  provide  a  body 
of  knowledge  encompassing  architectural  and  engineering  information  for  each  step  of  the 
acquisition  process.  In  addition  to  the  general  guidance,  NESI  contains  checklists  for  the  project 
manager  to  ensure  compliance  with  the  guidelines  set  forth  in  NESI  (US  Navy  Program 
Executive  Office  for  Command,  Control,  Communications,  Computers,  and  Intelligence,  2008). 
Unfortunately,  at  this  point,  compliance  with  NESI  is  not  required  by  the  Navy  or  any  DoD 
agency  (Eitelberg,  2008). 
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Chapter  3.  Methodology 


The  Joint  Force  Protection  Advanced  Security  System  (JFPASS)  has  the  unique 
challenge  of  creating  a  single,  joint  architecture  to  represent  force  protection  across  the  services. 
This  architecture  must  be  understood  by  various  stakeholders  as  well  as  represent  an  effective 
system,  which  will  be  the  groundwork  for  all  future  force  protection  acquisition  efforts.  This  led 
the  architecture  developers  to  seek  out  a  tool  for  evaluating  and  gaining  insight  into  their 
product. 

Drawing  upon  Value-Focused  Thinking  (VFT),  this  research  presents  a  Value-Driven 
Enterprise  Architecture  score  for  mapping  architectural  products  to  stakeholder  acquisition 
values.  The  generalized  VFT  methodology  laid  out  in  Chapter  2  is  built  upon  in  this  section  to 
extend  to  enterprise  architecture  evaluation  with  a  value  focus  in  application  to  the  JFPASS 
architecture.  Each  step  of  the  process  is  examined  in  depth  to  build  the  final  hierarchy.  This 
includes  all  steps  up  to  and  including  step  7. 

3.1  Problem  Identification 

The  JFPASS  project  grew  out  of  the  DoD’s  need  to  be  both  more  net-centric  and  more 
joint.  A  series  of  architectural  products  were  developed  to  meet  this  requirement.  This 
architecture  has  similar  problems  to  other  architectures  in  the  lack  of  effective  evaluation  tools, 
but  a  new  element  for  this  problem  is  the  extreme  complexity  of  the  architecture  and  the  desire  to 
examine  both  the  architectural  quality  and  the  System  Effectiveness  with  a  single  tool. 

To  solve  the  problem  of  evaluating  the  architecture,  the  research  question  was  framed  as: 
“How  should  common  Joint  Force  Protection  values  be  used  to  evaluate  a  “To-Be”  architecture 
for  net-centric  force  protection”  (Havlicek,  2008).  In  this  case,  the  system  is  the  JFPASS 
architecture.  To  further  define  the  problem,  the  context  was  first  researched  and  defined. 
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Several  documents  refer  to  the  protection  of  personnel,  assets,  and  information  as  the  three  key 
areas  to  be  protected  in  the  context  of  joint  force  protection  (IUBIP,  2006;  JCS,  2004;  Office  of 
the  CJCS,  2008;  Office  of  the  CJCS,  2007).  Figure  3.1  shows  a  hierarchy  of  the  documentation 
and  guidance  that  guides  force  protection.  Figures  3.2  through  3.4  specialize  this  idea  to  show 
the  mediums  from  which  each  area  must  be  protected. 


Figure  3.1.  Documentation  Hierarchy  for  Force  Protection 
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Protect  Personnel 


Figure  3.2.  Protect  Personnel  Specialization 


Figure  3.3.  Protect  Assets  Specialization 
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Figure  3.4.  Protect  Information  Specialization 


The  National  Security  Strategy  (NSS)  provides  the  highest-level  guidance  for  military 
operations  (Office  of  the  President  of  the  United  States  of  America,  2002).  The  National 
Military  Strategy  (NMS)  specifies  the  military  portion  of  the  NSS  (Joint  Chiefs  of  Staff,  2004). 
One  of  the  four  tenets  of  the  NMS  is  force  protection,  which  is  divided  into  the  protection  of 
personnel,  assets,  and  information.  The  IUBIP  specifies  that  the  scope  of  force  protection  be 
across  fixed,  semi-fixed,  and  mobile  sites.  Fixed  sites  are  defined  as  those  facilities  in  either  the 
Continental  United  States  (CONUS)  or  Outside  the  Continental  United  States  (OCONUS)  where 
Mutual  Security  Agreements  or  Status  of  Forces  Agreements  exist.  Semi-fixed  sites  are  any 
locations  established  for  a  temporary  purpose,  which  includes  expeditionary  locations  or 
locations  in  the  CONUS  or  OCONUS  that  are  no  intended  to  be  occupied  for  more  than  one  year 
at  construction.  Finally,  mobile  sites  are  those  where  a  unit  is  perfonning  its  mission,  including 
convoys,  logistics  patrols,  or  other  movements  between  sites. 
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At  each  of  the  three  types  of  sites,  personnel  and  assets  must  be  protected  in  the  same 
basic  ways,  as  the  personnel  generally  depend  on  the  assets  (vehicles  and  buildings)  for  shelter 
and  movement.  Often  attacking  an  asset  will  have  a  direct  affect  on  the  security  of  the  associated 
personnel.  Chemical,  Biological,  Radiological,  Nuclear,  and  High-Yield  Explosives  (CBRNE) 
risks,  however,  are  only  considered  for  personnel,  since  CBRNE  threats  do  not  have  an  effect  on 
the  assets,  aside  from  denial  of  their  use.  The  CBRNE  threat  is  intended  to  impact  the  personnel 
occupying  the  asset.  Protecting  information  is  a  slightly  different  concept.  Infonnation  must  be 
protected  in  two  contexts:  the  infrastructure  that  carries  information  and  the  access  that 
individuals  have  to  that  infonnation.  Access  control  involves  not  only  electronic  access  control 
such  as  ensuring  that  only  authorized  personnel  have  access  to  the  infonnation,  but  also  ensuring 
that  access  is  not  granted  from  person  to  person-to-mission  critical  infonnation. 

Within  the  context  of  protecting  personnel,  assets,  and  infonnation,  force  protection  must 
accomplish  all  of  the  DAWDR  (Detect,  Assess,  Warn,  Defend,  Recover)  activities.  For  the 
JFPASS  project,  the  focus  of  the  architecture  is  only  on  the  Detect,  Assess,  and  Warn  activities, 
although  mechanisms  exist  in  the  architecture  for  the  Defense  and  Recovery  of  a  location. 

3.2  Create  Value  Hierarchy 

The  first  decision  to  be  made  regarding  the  value  hierarchy  was  the  basic  split  of  how  to 
evaluate  the  system.  Two  divisions  of  quality  must  be  addressed:  the  quality  and  accuracy  of  the 
architectural  views  or  products  and  the  effectiveness  of  the  instantiated  system  that  the 
architectural  products  attempt  to  represent.  Due  to  the  complexity  of  the  system,  the  divisions 
were  separated  into  separate  branches.  This  decision  leads  to  better  decomposability  and  easier 
operability  (Kirkwood,  1997).  Splitting  the  architecture  from  the  system  also  ensures 
independence  and  mutual  exclusivity  of  each  value,  as  some  ideas  apply  to  each  side,  but  in  a 
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different  context.  Furthermore,  the  operability  of  the  hierarchy  is  improved  by  making  the 
hierarchy  not  only  easier  to  read,  but  also  extendable  to  other  situations.  With  the  architecture 
and  system  aspects  separate,  both  branches  may  be  used  independently  in  other  projects.  Either 
of  the  two  branches  may  also  be  replaced  by  another  branch  to  increase  accuracy  for  use  on 
another  system,  thus  making  the  hierarchy  modular.  However,  making  this  separation  violates 
the  desirable  property  “small  size”  of  a  hierarchy.  By  separating  the  two  values,  the  hierarchy 
would  potentially  be  larger  in  terms  of  total  branches,  although  the  number  of  measures  would 
stay  the  same  due  to  the  requirement  to  measure  the  same  infonnation,  regardless  of  the  outcome 
of  this  decision. 

3.2.1  Hierarchy  Background 

The  initial  value  hierarchy  created  to  address  the  problem  presented  in  Section  3.1  was 
developed  using  “ilities”  and  the  affinity  diagramming  process.  A  representative  list  of  ilities 
was  gathered  from  various  sources,  including  Ross  (2006);  McManus,  Richards,  Ross,  and 
Hastings  (2007);  and  INCOSE  (International  Council  on  Systems  Engineering,  2007).  The 
internet  website  “Wikipedia”  also  contains  a  list  of  ilities,  which  was  used  to  ensure  that  as  many 
ilities  as  possible  were  gathered.  Since  Wikipedia  is  a  user-edited  site,  this  gives  a  wider  pool  of 
quality  attributes  from  which  to  pull  at  the  risk  of  getting  inaccurate  infonnation.  Since  this  data 
pull  was  intended  only  to  gather  terms,  not  definitions  or  uses,  inaccurate  information  was  not  an 
issue  (Wikipedia,  2006).  This  search  for  ilities  was  done  in  place  of  an  on-site  brainstonning 
process,  to  ensure  that  previous  research  and  uses  of  the  quality  attributes  were  represented  in  the 
list.  The  full  list  of  98  ilities  is  shown  in  Appendix  A.  Ilities  were  chosen  for  this  exercise  for 
their  historical  use  in  describing  the  quality  attributes  of  various  systems.  By  finding  all  of  the 
applicable  ilities  related  to  the  project,  it  was  possible  to  capture  all  of  the  necessary  quality 
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attributes  to  describe  both  the  architectural  quality  of  the  products  and  the  effectiveness  of  the 
instantiated  system. 

In  accordance  with  the  affinity  diagramming  process,  each  of  these  ilities  was  written  on 
individual  note  cards.  The  affinity-diagramming  team  was  then  sequestered  in  silence  to 
physically  arrange  the  note  cards  into  groups  of  similar  qualities.  At  the  conclusion  of  the 
process,  30  subgroups  were  found.  This  led  to  an  interactive  discussion  of  the  groupings  and 
further  refinement  of  the  subgroups.  This  discussion  first  identified  quality  attributes  and 
subgroups  that  did  not  apply  to  the  JFPASS  project.  The  eliminated  ilities  were:  composability, 
demonstrability,  leamability,  nomadicity,  portability,  predictability,  seamlessness,  testability, 
timeliness,  trainability,  and  transactionality.  Composability  refers  to  creating  some  new  form  by 
combining  components.  While  the  construction  of  the  system  will  be  created  through  the 
combination  of  its  components,  the  ability  to  do  so  was  not  considered  a  measurable  quality 
attribute.  The  actual  combination  of  components  is  a  design  consideration  that  must  be 
considered  before  any  architectural  products  are  produced.  Demonstrability  and  testability  refer 
to  the  ability  of  the  system  to  be  demonstrated.  These  ilities  were  eliminated  because  the 
JFPASS  Joint  Capabilities  Technology  Demonstration  is  already  in  progress;  therefore,  the 
demonstrability  of  the  system  is  assumed  and  is  not  required  as  a  quality  attribute.  Leamability 
and  trainability  were  considered  too  vague  due  to  the  confusion  as  to  whether  the  learning 
referred  to  the  system  or  the  system  users.  Trainability  is  an  attribute  that  will  be  considered  at 
some  point  in  the  creation  of  the  system,  but  as  JFPASS  is  a  “system  of  systems”  still  in  the 
design  phase,  there  is  no  way  to  definitively  measure  the  ability  of  the  system  to  be  taught  to 
others.  Nomadicity  and  portability  were  eliminated  because  the  JFPASS  is  not  considered  a 
“mobile”  system.  Predictability  and  seamlessness  were  also  vague  in  definition  in  terms  of  the 
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JFPASS  system.  Timeliness  refers  to  the  time  required  to  create  the  system.  While  timeliness 
could  be  defined  to  show  when  the  system  will  be  fielded  and  implemented,  there  are  currently 
no  time  estimates  or  requirements  for  the  fielding  of  the  system.  Finally,  transactionality  was 
eliminated  due  to  the  connotation  of  its  root  word,  “transaction.”  There  will  be  no  monetary 
transactions  taking  place  as  a  function  of  this  system;  therefore,  the  ility  was  eliminated. 

The  resulting  22  subgroups  were  then  examined  for  agreement  and  accuracy.  After 
minor  alterations  to  the  locations  of  certain  words  based  on  definition,  super  groups  were 
formed.  The  two  primary  supergroups  were  based  on  the  decision  to  split  the  architecture  and 
system  qualities.  The  groups  related  to  the  quality  of  the  system  being  represented  were  placed 
in  a  supergroup  called  “ System  Effectiveness ,”  while  the  groups  related  to  the  quality  of  the 
architectural  products  were  placed  in  a  supergroup  called  “ Architecture  Quality .”  Architecture 
Quality  is  addressed  by  Cotton  and  Haase  (2009).  The  System  Effectiveness  supergroup  is 
addressed  here.  The  group  names  shown  in  the  following  discussion  and  tables  are  based  on  the 
final  decisions,  following  sponsor  discussion. 

The  System  Effectiveness  value  was  subsequently  decomposed  into  three  second-tier 
values:  Capability,  Maintainability,  and  Interoperability.  Maintainability  was  comprised  of  two 
third-tier  values  called  Dependability  and  Resiliency.  Dependability  was  further  decomposed 
into  Supportability  and  Reliability,  while  Resiliency  was  decomposed  into  Survivability  and 
Recoverability.  Appendix  B  shows  the  final  System  Effectiveness  quality  attributes  grouped 
according  to  value,  along  with  their  synonyms. 

Prior  to  creating  the  hierarchy  itself,  each  value  group  name  was  defined  to  ensure  that 
the  synonyms  that  compose  the  group  were  accounted  for  in  the  final  consideration  of  the  group. 
Each  of  the  value  definitions  are  listed  in  Table  3.2.  These  definitions  incorporated  the 
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definition  of  the  quality  attribute  itself  and  the  synonyms  for  the  particular  group  name,  as  well 
as  the  value  to  the  decision-maker.  Following  sponsor  discussion,  the  defined  and  grouped  ilities 
were  converted  into  a  value  hierarchy.  The  resulting  hierarchy  is  shown  in  Figure  3.5 


Table  3.2  System  Effectiveness  Value  Definitions 


System  BhcSwn—  Values 

Value  Delinitiori 

CapaMty 

A  system's  ability  to  produce  the  expected  or  desired  results  on  the  battlefield. 

PnposeMms 

The  ability  of  a  system  to  address  the  problem  whidi  it  is  intended  to  solve.  The  relevance  of  a  system  in  a  given 
context  or  situation. 

PiacticaEty 

The  system's  ability  to  be  achieved  within  realistic  constraints,  including  economic,  constructability,  and  timeliness. 

FfenbOty 

The  ability  of  a  system  to  be  changed  based  on  Operational  need.  This  changeability  refers  to  its  ability  to  be  altered 
before,  during  and  after  a  conflict. 

MantamafaAty 

A  system's  ability  to  be  kept  at  its  intended  level  of  operation . 

Dependafaftty 

A  system's  ability  to  continue  operating  at  its  intended  standard. 

Siqiatahity 

The  ability  of  a  system  to  be  realistically  sustained  and  remain  functional  and  useful  given  the  expenditure  of  a 
reasonable  amount  of  effort. 

RefiafaOty 

The  ability  of  a  system  to  perform  as  intended  and  execute  given  functions  if  properly  maintained  and  supported. 

ResSency 

A  system's  ability  to  be  returned  to  its  intended  standard. 

SwivabSty 

The  ability  to  survive  attack  or  other  enemy  action  and  continue  to  operate  as  originally  intended  or  retain  the  ability 
of  being  repaired  and  restored  to  operational  status. 

Recoverafafity 

The  system's  ability  to  be  repaired  or  recovered  following  an  attack  or  other  damage  within  an  allotted  time  frame. 

A  system's  ability  to  be  applied  within  different  contexts,  including  other  services  and  organizations. 

InterdiangeaMrty 

The  ability  of  parts,  components,  systems,  units,  and  people  to  be  substituted  across  organizations  and  systems 
within  the  system  of  systems. 

Comm 

The  system's  ability  to  transmit  information  in  timely  and  accurate  way  as  to  facilitate  analysis,  derision  making,  and 
derisive  action. 

62 


63 


Figure  3.5.  System  Effectiveness  Value  Flierarchy 


3.2.2  System  Effectiveness  Hierarchy 

In  the  creation  of  the  value  hierarchy,  several  naming  changes,  definition  changes,  and 
alterations  to  structure  were  required  to  create  a  representative  hierarchy.  These  changes  were 
accomplished  through  further  literature  review  and  during  meetings  with  the  decision-maker  and 
sponsoring  organization.  The  value  hierarchy  was  created  using  the  resulting  groups  of  the 
affinity  diagramming  process,  based  on  quality  attributes.  This  section  describes  the  values 
found  on  each  level  of  the  hierarchy  in  more  depth.  The  initial  draft  hierarchy  was  presented 
during  these  meetings,  resulting  is  discussion  and  ultimately  validation  by  a  panel  of  Subject 
Matter  Experts  (SMEs)  and  the  decision-maker. 

In  the  initial  iteration  of  the  value  tree,  the  System  Effectiveness  branch  was  called 
System  Value.  During  a  discussion  regarding  the  naming  and  definition  of  the  Capability  value 
(which  was  initially  named  Effectiveness),  it  was  decided  that  the  term  Effectiveness  better 
described  the  entire  System  branch  as  opposed  to  a  single  value  under  the  system  branch; 
therefore,  Effectiveness  was  promoted  to  the  branch  name  to  better  describe  all  of  the  values 
under  the  branch.  Each  of  the  values  in  the  branch  ( Capability ,  Maintainability,  and 
Interoperability)  relate  to  the  overall  effectiveness  of  the  instantiated  system. 

3.2.2. 1  Capability  Branch 

Capability  was  originally  known  as  Effectiveness.  However,  Effectiveness  was  decided 
to  be  too  broad  of  a  term  to  describe  the  purpose  of  the  Capability  branch.  The  Capability 
branch  was  defined  as,  “A  System’s  ability  to  produce  the  expected  or  desired  results  on  the 
battlefield.”  Subsequently,  it  is  intended  to  represent  the  operational  ability  of  the  system  to 
accomplish  its  intended  purpose.  In  other  words,  a  system  must  have  the  ability  to  meet  the 
objectives  for  which  it  was  designed.  On  the  third  tier  of  the  hierarchy,  Capability  is 
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decomposed  into  Purposefulness,  Practicality,  and  Flexibility.  These  lower  tier  values  serve  to 
specify  the  ideas  that  make  up  Capability. 

Within  the  context  of  Capability,  the  “the  ability  of  a  system  to  address  the  problem 
which  it  is  intended  to  solve  or  the  relevance  of  a  system  in  a  given  context  or  situation”  was 
called  the  Purposefulness  of  the  system.  This  value  actually  defines  whether  the  system  does 
what  it  is  intended  to  do  in  the  proper  situations.  The  value  of  Purposefulness  is  used  to  account 
for  a  major  idea  of  System  Effectiveness.  This  value  accounts  for  a  great  deal  of  the  Capability 
of  a  system. 

The  Practicality  of  the  system  defines  whether  it  can  actually  be  realized.  Practicality  is 
officially  defined  as,  “The  system’s  ability  to  be  achieved  within  realistic  constraints,  including 
economic,  constructability,  and  timeliness.”  This  value  was  considered  important  due  to  the 
inability  of  the  system  to  accomplish  its  intended  objectives  if  it  cannot  be  constructed  or 
implemented  within  realistic  constraints.  Without  a  practical  system,  the  goals  of  the  system 
designer  will  not  be  achieved. 

The  Flexibility  value  was  initially  placed  under  the  Interoperability  branch  due  to  a 
connotation  involving  its  ability  to  change  to  operate  with  other  systems.  However,  a  system’s 
ability  to  change  in  relation  to  other  systems  is  primarily  detennined  by  its  initial  design  once  it 
is  implemented.  Therefore,  Flexibility  was  moved  to  better  capture  the  system’s  ability  to 
change  to  meet  changing  and  evolving  operational  objectives.  This  allowed  a  more  strict 
definition,  which  eliminated  the  problem  of  broad  connotations  for  the  word  Flexibility.  The 
official  definition  of  Flexibility  is  “the  ability  of  a  system  to  be  changed  based  on  operational 
need.  This  changeability  refers  to  its  ability  to  be  altered  before,  during,  and  after  a  conflict.” 
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3.2.2.2  Maintainability  Branch 

The  entire  Maintainability  branch  was  validated  by  the  decision-maker  as  being 
acceptable  as  initially  presented.  Definitions  of  values  were  altered  slightly  to  ensure  maximum 
achievement  of  the  decision-maker’s  values.  Maintainability  itself  is  defined  as,  “A  system’s 
ability  to  be  kept  at  its  intended  level  of  operation.”  Any  system  requires  some  type  of  regular 
action  to  ensure  that  its  intended  operation  continues  uninterrupted. 

Below  Maintainability,  the  first  branch  was  called  Dependability.  Dependability  is 
officially  defined  as,  “A  system’s  ability  to  continue  operating  at  its  intended  standard.”  The 
Dependability  branch  refers  to  the  maintenance  of  the  system  under  normal  operating  conditions. 
Dependability  deals  with  the  “peacetime”  operations  and  maintenance  of  a  system.  On  the  next 
tier,  Supportability  is  one  of  the  values  under  Dependability.  This  value  deals  with  a  system’s 
ability  to  operate  as  normal,  given  a  standard  maintenance  schedule.  Dependability's  definition 
is  “the  ability  of  a  system  to  be  realistically  sustained  and  remain  functional  and  useful  given  the 
expenditure  of  a  reasonable  amount  of  effort.”  A  reasonable  amount  of  effort  refers  to 
operations  performed  in  accordance  with  a  standard  maintenance  schedule.  This  value  does  not 
incorporate  major  alterations  or  repairs,  only  a  normal  recurring  work  program  type  of 
maintenance.  The  second  value  under  Dependability  is  Reliability.  The  Reliability  value  deals 
with  the  ability  of  the  system  to  continue  its  operation  if  maintained  properly.  Reliability  is  the 
relationship  between  Supportability  and  the  operation  of  the  system.  Its  definition  is  “the  ability 
of  a  system  to  perform  as  intended  and  execute  given  functions  if  properly  maintained  and 
supported.” 

Resiliency  is  the  second  branch  falling  directly  under  Maintainability.  Resiliency  is  “a 
system’s  ability  to  be  returned  to  its  intended  standard.”  If  the  Dependability  branch  deals  with  a 
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system’s  operation  during  normal  peacetime  or  uninterrupted  operations,  Resiliency  refers  to  the 
system’s  operation  following  some  type  of  interruption.  In  the  context  of  Joint  Force  Protection, 
this  interruption  is  some  type  of  hostile  action.  Resiliency  measures  how  easily  a  system  may  be 
repaired  following  such  an  action.  Under  Resiliency,  the  first  of  two  values  is  Survivability. 
Survivability  is  the  part  of  Resiliency  dealing  with  a  component’s  ability  to  withstand  some 
hostile  action.  It  is  defined  as  “the  ability  to  survive  attack  or  other  enemy  action  and  continue  to 
operate  as  originally  intended  or  retain  the  ability  of  being  repaired  and  restored  to  operational 
status.”  Survivability  measures  how  a  system  operates  once  it  has  been  affected  by  some  hostile 
action.  The  second  Resiliency  sub-value  is  Recoverability.  Recoverability  is  another  portion  of 
Resiliency  referring  to  a  system’s  ability  to  be  returned  to  full  operational  status  following  an 
interruption  of  operations  due  to  hostile  action.  It  is  defined  as  “the  system’s  ability  to  be 
repaired  or  recovered  following  an  attack  or  other  damage  within  an  allotted  time  frame.”  The 
definition  refers  to  repair  and  recovery,  both  of  which  are  intended  to  allude  to  the  returning  of 
the  system  to  its  original  intended  operation  or  the  state  that  it  was  at  prior  to  the  hostile 
interruption.  “An  allotted  time  frame”  is  a  time  period  at  the  decision-maker  or  user’s  discretion. 
Any  system  or  system  component  must  be  designed  to  be  returned  to  its  original  level  of 
operation  within  a  specified  time  frame. 

3.2.2.3  Interoperability 

Interoperability  is  the  value  which  measures  the  ability  of  the  system  to  operate  in 
conjunction  with  other  systems  and  nodes.  Interoperability  covers  both  the  net-centricity  of  a 
system  and  its  ability  to  be  used  in  different  contexts.  Interoperability  is  defined  as  “a  system’s 
ability  to  be  applied  within  different  contexts,  including  other  services  and  organizations.” 
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In  terchangeability  is  the  portion  of  In  teroperability  that  accounts  for  the  system  and 
components  to  be  useful  across  different  contexts.  The  decision-maker  and  subject  matter 
experts  felt  that  it  was  very  important  for  components  to  be  interchangeable.  Each  system  and 
component  should  be  able  to  be  changed  out  for  another  seamlessly.  This  concept  includes 
personnel  as  well  as  components.  Soldiers,  Airmen,  Sailors,  and  Marines  must  all  have  the  same 
basic  knowledge  on  the  systems  in  question  and  be  trained  to  the  same  level  in  force  protection 
awareness.  JOINT  OPERATIONS  are  extremely  important  and  to  accomplish  a  truly  Joint 
environment,  all  personnel  must  have  a  similar  training  base.  Interchangeability  is  defined  as  “a 
system’s  ability  to  be  applied  within  different  contexts,  including  other  services  and 
organizations.” 

Communication  refers  to  the  ability  of  the  nodes  within  the  system  to  communicate  with 
each  other.  Both  infrastructure  and  common  languages  are  important  for  this  value  to  be 
achieved.  Communication  is  “the  system’s  ability  to  transmit  infonnation  in  timely  and  accurate 
way  as  to  facilitate  analysis,  decision  making,  and  decisive  action.”  An  interoperable  system 
must  send  information  between  nodes  quickly  and  with  complete  data  integrity. 

3.3  Develop  Evaluation  Measures 

With  the  full  value  hierarchy  built,  each  lowest-tier  value  must  be  measured.  Therefore, 
the  lowest-tier  values  were  assigned  one  or  multiple  evaluation  measures.  The  goal  of  an 
evaluation  measure  is  to  detennine  the  level  of  attaimnent  of  each  value.  Table  3.3  lists  all 
evaluation  measures,  including  their  important  characteristics.  Although  weights  are  not 
discussed  until  section  3.3,  Table  3.3  also  shows  the  global  weights  (1)  for  each  measure.  The 
source  for  each  measure  suggests  where  a  scorer  should  start  investigating  the  architecture  to 
find  the  information.  Table  3.4  presents  the  definitions  for  each  measure.  Given  the  vast  range 
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Table  3.3.  System  Effectiveness  Evaluation  Measures 


Measure  Name 

m 

Type 

Source1 

Lower 

Bound 

Upper 

Bound 

1 

OPERATIONAL  NEEDS 

0.041 

Constructed  - 
Direct 

SV-5 

0% 

100% 

2 

THREAT  DETECTION 

0.041 

Constructed  - 
Proxy 

OV-5 

No 

Yes 

3 

THREAT  ASSESSMENT 

0.041 

Constructed  - 
Proxy 

OV-5 

No 

Yes 

4 

WARNING  PLAN 

0.041 

Constructed  - 
Proxy 

OV-5 

No 

Yes 

5 

TECHNOLOGICAL 

AVAILABILITY 

0.02 

Natural  - 
Direct 

SV-7,8,9 

TRL  1 

TRL  9 

6 

ENVIRONMENTAL  IMPACT 

0.02 

Constructed  - 
Proxy 

AV-1 

Cannot  be 
built 

Within  all 
constraints 

B 

MONETARY  PRACTICALITY  - 
INITIAL 

0.02 

Natural  - 
Direct 

AV-1 

Over 

budget 

Under  budget 

8 

MONETARY  PRACTICALITY  - 
MAINTENANCE 

0.02 

Natural  - 
Direct 

AV-1 

Over 

budget 

Under  budget 

9 

ADAPTATION 

0.027 

Constructed  - 
Proxy 

SV-8 

Static 

Easy,  On-Site 

10 

SUPPORT  ABILITY 
REQUIREMENTS 

0.035 

Constructed  - 
Direct 

SV-7 

No 

Yes 

11 

RELIABILITY  REQUIREMENTS 

0.064 

Constructed  - 
Proxy 

SV-7 

No 

Yes 

12 

SYSTEM  REDUNDANCY 

0.04 

Constructed  - 
Direct 

OV-6 

None 

All/Multiple 

13 

RECOVERABILITY 

REQUIREMENTS 

0.026 

Constructed  - 
Direct 

SV-7 

No 

Yes 

14 

JOINT  OPERATIONS 

0.033 

Constructed  - 
Proxy 

AV-1 

No 

Yes 

15 

NESI  DEVELOPMENT 

0.066 

Constructed  - 
Direct 

TV-1 

No 

Yes 

16 

NESI  EVALUATION 

0.066 

Constructed  - 
Proxy 

TV-1 

No 

Yes 

Total  of  System  Effectiveness  Global 

Weights 

0.600 

1.  Primary  source  of  information;  other  views  may  be  required 
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Table  3.4.  Measure  Definitions 


System  Effectiveness  Branch 

Value 

Measure 

Measure  Definition 

Purposefulness 

OPERATIONAL 

NEEDS 

What  percentage  of  operational  needs  are  addressed 
by  the  system? 

Purposefulness 

THREAT 

DETECTION 

Has  a  Threat  Detection  Plan  been  established? 

Purposefulness 

THREAT 

ASSESSMENT 

Has  a  Threat  Assessment  Plan  been  established? 

Purposefulness 

WARNING  PLAN 

Has  a  base  warning  plan  been  established? 

Practicality 

TECHNOLOGICAL 

AVAILABILITY 

What  is  the  Technological  Availability  of  the 
system? 

Practicality 

ENVIRONMENTAL 

IMPACT 

Can  the  system  be  realized  within  Environmental 
Constraints? 

Practicality 

MONETARY 
PRACTICALITY  - 
INITIAL 

Can  the  system’s  initial  cost  be  realized  within 
current  budgetary  constraints? 

Practicality 

MONETARY 
PRACTICALITY  - 
MAINTENANCE 

Can  the  system  be  maintained  within  current 
budgetary  constraints? 

Flexibility 

ADAPTATION 

How  well  does  the  system  adapt  to  changing  threats? 

Supportability 

SUPPORTABILITY 

REQUIREMENTS 

Have  supportability  requirements  been  accounted 
for? 

Reliability 

RELIABILITY 

REQUIREMENTS 

Have  reliability  requirements  been  accounted  for? 

Survivability 

SYSTEM 

REDUNDANCY 

The  degree  to  which  critical  systems  are  redundant? 

Recoverability 

RECOVERABILITY 

REQUIREMENTS 

Have  recoverability  requirements  been  accounted 
for? 

In  terchangeabi  lity 

JOINT 

OPERATIONS 

Have  CONOPs  been  constructed  to  account  for  all 
organizations? 

Communication 

NESI 

DEVELOPMENT 

Was  NESI  Guidance  taken  into  account  when 
constructing  architecture? 

Communication 

NESI  EVALUATION 

Has  a  NESI  evaluation  been  completed  on  the 
architecture? 
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of  available  products  and  latitude  for  which  architects  may  use  the  products,  it  may  be  necessary 
to  examine  other  products  or  views  to  find  the  information  necessary  for  each  measure.  In  cases 
where  a  specific  view  has  been  created  for  the  sole  purpose  of  representing  a  certain  type  of 
information,  that  view  will  be  considered  essential  in  the  measurement  of  the  value  at  hand.  This 
accounts  for  information  that  may  be  extrapolated  by  the  user  examining  several  other  products, 
but  was  not  explicitly  stated  by  the  architect,  when  it  is  possible. 

3.3.1  Capability  Measures 

The  goals  of  the  measures  under  Capability  are  to  collectively  measure  the  attainment  of 
the  Capability  value.  Capability  is  meant  to  determine  if  the  system  is  able  to  produce  the 
sponsor’s  desired  effects  on  the  battlefield.  Through  use  of  the  Purposefulness,  Practicality,  and 
Flexibility  values,  the  three  major  aspects  of  Capability  are  captured.  Each  of  these  lower  tier 
values  must  be  measured  to  determine  their  effect  on  the  Capability  value. 

3.3. 1.1  Evaluation  Measures  for  Purposefulness 

Purposefulness  requires  more  than  one  measure  to  completely  detennine  its  level  of 
attainment.  Four  total  measures  score  four  separate  aspects  of  Purposefulness  as  detennined  by 
the  sponsor.  Each  of  these  measures  serves  to  detennine  a  particular  portion  of  the 
Purposefulness  value.  This  is  where  the  Detect,  Assess,  Warn,  Defend,  Recover  (DAWDR) 
construct  was  considered. 

3.3. 1.1.1  OPERATIONAL  NEEDS 

The  first  evaluation  measure  under  the  Purposefulness  value  determines  whether  the 
system  designers  have  accounted  for  the  sponsor’s  initial  requirements.  The  measure  asks: 

“What  percentage  of  the  Operational  Needs  is  addressed  by  the  system?”  It  is  possible  to  trace 
each  operational  need  to  a  requirement,  to  a  capability,  and  ultimately  down  to  a  function  or 
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system  capability.  The  DoDAF  facilitates  this  primarily  through  the  SV-5  product.  The  SV-5 
alone  is  not  capable  of  accomplishing  this  measurement,  due  to  its  lack  of  an  explicit  list  of 
OPERATIONAL  NEEDS.  The  AV-1  product  must  be  used  in  coordination  with  the  SV-5  to  find 
this  association.  This  measure  is  Constructed-Direct.  The  SV-5  is  a  constructed  product, 
through  which  the  scorer  may  determine  if  OPERATIONAL  NEEDS  have  been  met  by  system 
components.  The  measurement  scale  is  not  a  common  way  of  determining  whether 
OPERATIONAL  NEEDS  have  been  met,  as  SV-5s  require  some  architecture  knowledge  to  read.  It 
directly  measures  the  value  due  to  the  fact  that  detennining  if  all  operational  needs  have  been 
accounted  for  determines  the  exact  level  of  attainment  of  the  value. 

3.3.1. 1.2  THREAT  DETECTION 

The  THREAT  DETECTION  measure  grew  out  of  the  DAWDR  (Detect,  Assess,  Warn, 
Defend,  Recover)  construct.  For  the  system  to  meet  its  purpose  and  accomplish  its  goals  on  the 
battlefield,  it  must  account  for  at  least  the  first  three  aspects  of  the  DAWDR  construct.  THREAT 
DETECTION  measures  whether  a  Threat  Detection  Plan  has  been  developed.  At  this  stage  of 
development,  it  is  impossible  to  measure  the  actual  quality  of  the  Threat  Detection  Plan;  it  is 
only  possible  to  measure  its  existence.  At  a  later  time  in  system  development,  it  may  be  possible 
to  alter  this  measure  to  account  for  the  quality  of  the  Threat  Detection  Plan.  The  activities 
associated  with  the  Threat  Detection  Plan  will  be  located  in  the  OV-5  if  it  is  available.  By 
tracing  the  system’s  activities,  it  will  be  possible  to  detennine  if  there  are  system  activities  that 
account  for  threat  detection.  If  a  Threat  Detection  Plan  exists,  the  activities  that  accomplish  it 
must  also  be  present  in  the  OV-5.  If  the  OV-5  is  not  available,  the  OV-1  and  OV-3  will  be 
examined  for  a  detection  plan.  This  measure  is  a  Constructed-Proxy  measure.  It  will  be  obvious 
to  any  reader  if  the  plan  exists,  making  the  scoring  scale  binary.  The  measurement  question, 
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however,  was  constructed  for  the  purpose  of  this  evaluation.  By  using  the  activities  associated 
with  the  plan  as  opposed  to  the  plan  itself,  the  scorer  must  make  an  inference  that  if  there  are 
activities,  then  the  plan  has  been  created;  therefore,  it  is  a  proxy  measure. 

3 .3. 1.1.3  THREAT  ASSESSMENT 

THREAT  ASSESSMENT  is  similar  in  all  ways  to  THREAT  DETECTION,  except  it  measures 
whether  a  plan  exists  to  assess  the  threat  once  it  has  been  detected.  This  accounts  for  the  second 
aspect  of  the  DAWDR  construct.  This  measure  will  also  examine  the  OV-5  to  look  for  activities 
related  to  Assessment  (with  OV-1  and  OV-3  as  secondary  views).  THREAT  ASSESSMENT  is  a 
Constructed-Proxy  measure. 

3.3. 1.1.4  WARNING  PLAN 

WARNING  PLAN  is  the  third  aspect  of  the  DAWDR  construct  measured  by  this  model.  It 
is  also  very  similar  to  THREAT  DETECTION  and  THREAT  ASSESSMENT.  In  this  case  though,  the 
OV-5  is  examined  for  any  mentions  of  the  activities  related  to  warning  the  base  population.  A 
warning  plan  should  warn  not  only  the  base  population,  but  also  specific  organizations  as 
required.  Warnings  may  also  be  “tiered,”  so  that  only  certain  people  are  warned  depending  on 
the  purpose  for  the  warning.  Warning  plans  may  be  very  prevalent  in  the  OV-1  and  OV-3  in  this 
case,  as  the  warning  of  people  relates  directly  to  both  the  system  boundary  and  node 
communication.  It  is  also  Constructed-Proxy. 

3.3. 1.2  Evaluation  Measures  for  Practicality 

The  value  of  Practicality  must  also  be  measured  using  more  than  one  aspect.  Since  it  is 
also  a  very  complex  issue  and  the  definition  calls  for  several  layers  of  practicality,  each  of  those 
layers  must  be  measured.  Each  of  the  measures  of  Practicality  measures  an  idea  that  was 
declared  to  be  an  important  measure  by  the  decision-maker. 
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3.3. 1.2.1  TECHNOLOGICAL  AVAILABILITY 


Technology  Readiness  Levels  (TRLs)  are  described  in  the  Defense  Acquisition 
Guidebook  (Department  of  Defense,  2004).  The  nine  Technology  Readiness  Levels  (TRL)  were 
created  by  the  U.S.  government  to  “assess  the  maturity  of  evolving  technologies.”  They  allow 
the  decision-maker  to  detennine  the  level  of  development  of  certain  systems  during  the 
acquisition  process.  TRLs  are  being  used  in  this  case  to  measure  the  overall  Technological 
Availability  of  the  system  of  systems.  The  TRL  of  each  component  will  be  averaged  to 
detennine  an  overall  TRL  for  the  entire  system.  TRLs  measure  Practicality  by  detennining  how 
easy  it  will  be  to  actually  construct  the  system.  They  are  capable  of  showing  whether  a 
technology  is  still  in  the  very  early  development  stages  or  available  immediately  off  the  shelf. 
This  is  a  Natural-Direct  measure  since  TRLs  are  widely  used  within  the  DoD  and  are  intended  to 
measure  the  Technological  Availability  of  systems  prior  to  acquisition.  Each  of  the  nine  levels  is 
detailed  in  Section  3.4.  The  SV-9  is  the  most  likely  location  for  TRL  information.  The  system 
developers  have  latitude  in  detennining  exactly  which  view  will  provide  the  TRLs  for  each 
component.  It  is  possible  for  the  AV-1  to  also  have  a  more  direct  assessment  of  the  TRL  of  the 
entire  system. 

3.3. 1.2.2  ENVIRONMENTAL  IMPACT 

The  ENVIRONMENTAL  IMPACT  measure  detennines  if  the  system  can  be  realized  within 
the  environmental  constraints  of  a  given  location.  Since  the  system  is  not  in  any  specific 
location  during  the  design  phase,  it  is  examined  prior  to  construction  to  detennine  if  it  will  fit 
into  the  environmental  constraints  of  a  given  area.  For  example,  to  protect  the  perimeter  of  an 
installation,  the  base  could  build  a  20-foot  high,  10-foot  thick  concrete  wall  around  the 
installation  and  cover  the  entire  installation  with  netting  to  create  a  protective  “bubble,” 
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however,  this  method  would  be  cost,  time,  and  environmentally  restrictive.  Since  many 
environmental  laws  would  be  broken  in  order  to  accomplish  this  feat,  it  is  considered 
impractical.  Most  projects  require  some  sort  of  Environmental  Impact  Statement  (EIS)  before 
implementation,  but  at  this  level  of  development,  an  EIS  may  not  be  available.  It  would  be 
expected  that  language  regarding  the  environmental  practicality  would  be  included  in  the  AV-1 
of  a  systems  architecture,  but  any  environmental  specification  being  adhered  to  will  be  located  in 
the  TV-1;  therefore,  the  TV-1  is  the  primary  location  for  this  measurement.  This  measure  is  a 
Constructed-Proxy  measure.  This  infonnation  is  typically  included  in  an  EIS,  but  it  is  time 
restrictive  for  an  architecture  evaluator  to  do  this.  Environmental  Practicality  is  found  by 
examining  the  technical  standards  for  environmental  wording.  At  a  later  stage  of  development, 
this  measure  will  no  longer  be  Constructed-Proxy,  once  an  EIS  is  available  for  review. 

3.3. 1.2.3  MONETARY  PRACTICALITY  -  INITIAL 

In  the  previous  example  of  perimeter  detection,  the  large  concrete  wall  could  not  be 
constructed  due  to  its  impact  on  the  environment.  For  this  measure,  the  same  concept  applies, 
but  to  the  cost  of  the  wall.  The  MONETARY  PRACTICALITY  -  INITIAL  measure  detennines  if  the 
program  can  afford  the  initial  implementation  cost  of  a  project.  In  order  for  the  project  to  be 
completed  within  the  DoD,  funds  must  be  realistic  and  available.  The  Defense  Acquisition 
Guidebook  includes  specific  guidelines  on  funding  and  affordability  (Department  of  Defense, 
2004).  There  are  several  places  within  the  architecture  where  costs  may  be  incorporated.  For 
this  evaluation,  the  AV-1  will  first  be  examined  as  it  contains  overall  system  infonnation  and 
more  broad  concepts  such  as  cost.  If  information  is  not  found  there,  other  views,  such  as  the 
OV-5,  which  has  an  optional  “cost  overlay”  function,  will  be  examined.  This  is  a  Natural-Direct 
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measure  since  dollars  are  a  common  unit  of  measurement  and  the  estimate  directly  measures  the 
cost  of  the  project. 

3.3. 1.2.4  MONETARY  PRACTICALITY  -  MAINTENANCE 

This  measure  examines  the  “life-cycle  cost”  of  the  system.  Within  the  DoD  acquisition 
system,  both  initial  costs  and  life-cycle  costs  are  required  for  a  project  to  reach  milestones  within 
the  process  (Department  of  Defense,  2004).  Life-cycle  costs  provide  an  estimate  of  how  much  a 
system  will  cost  to  maintain  throughout  the  entire  effective  life  of  the  system,  including  disposal 
at  the  end  of  its  effective  life.  While  it  is  difficult  to  detennine  monetary  constraints  in  the 
future,  it  is  possible  to  examine  future  budgets  and  multi-year  plans  to  see  if  the  gaining  office 
can  work  the  additional  cost  into  their  projected  budgets.  This  measure  will  also  be  located 
primarily  in  the  AV-1  product.  MONETARY  PRACTICALITY  -  MAINTENANCE  is  also  a  Natural- 
Direct  measure  for  the  same  reasons  as  MONETARY  PRACTICALITY  -  INITIAL. 

3.3. 1.3  Evaluation  Measure  for  Flexibility 

Measuring  a  system’s  Flexibility  is  a  difficult  concept  and  requires  very  strict  definition 
of  the  value.  Flexibility  is  defined  based  on  its  ability  to  adapt  to  changes  on  the  battlefield; 
therefore,  the  measurement  seeks  to  detennine  its  ability  to  do  that.  The  Flexibility  value  has 
only  one  measure  of  effectiveness. 

A  system’s  Flexibility  is  directly  dependent  on  how  easy  or  possible  it  is  to  change  the 
system  configuration  quickly  and  effectively.  ADAPTION  measures  if  it  is  possible  to  change  the 
system  and  the  considerations  taken  to  do  so.  The  location  of  infonnation  such  as  this  is  not 
explicitly  stated  anywhere  in  the  DoDAF  views.  The  first  candidate  view  for  finding  this 
information  is  the  SV-8,  Systems  Evolution  Description.  The  SV-8  tells  the  reader  of  any 
planned  future  improvements  to  the  system  or  whether  adaptations  are  possible.  This  view 
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should  provide  information  as  to  how  easy  it  will  be  to  alter  the  system  as  well.  ADAPTION  is  a 
Constructed-Proxy  measure.  Since  there  is  no  standard  system  for  measuring  a  system’s 
flexibility,  a  scale  must  be  constructed  to  measure  this  concept.  Since  the  value  is  abstract, 
though  well  defined,  no  direct  measurement  will  be  possible;  it  is  a  proxy  measurement  of  things 
that  would  make  a  system  flexible. 

3.3.2  Maintainability  Measures 

In  discussions  with  the  decision-maker,  it  was  detennined  that  the  Maintainability  of  a 
system  is  often  absent  from  an  architecture.  Although  there  are  no  DoDAF  views  that 
specifically  require  explicit  infonnation  regarding  how  easy  a  system  is  to  maintain,  the  DoDAF 
does  provide  a  good  vehicle  for  doing  this.  The  SV-7  product  provides  the  perfonnance 
characteristics  for  the  components  in  the  system.  Therefore,  the  evaluator  must  extrapolate  this 
infonnation  from  the  SV-7.  There  are  obviously  certain  parameters  to  which  the  system  is 
designed.  Such  systems  engineering  concepts  as  “Mean  Time-Between-Failures  (MTBF),” 
“System  Availability,”  “Mean  Time-To-Repair,”  “Mean  Uptime,”  “Mean  Downtime,”  and 
“Reliability”  have  actual  equations  and  methods  for  evaluation.  These  things  are  normally 
specified  in  the  project  requirements,  i.e.,  the  system  must  be  designed  to  meet  certain  reliability 
standards.  For  these  things  to  be  designed  into  the  system,  design  calculations  must  be  done  at 
some  point  and  the  standard  must  be  included  in  the  architecture.  The  SV-7  is  the  view  normally 
associated  with  such  performance  standards,  but  these  characteristics  are  typically  not  included 
in  the  SV-7.  The  measures  for  Maintainability  are  SUPPORTABILITY  REQUIREMENTS, 
RELIABILITY  REQUIREMENTS,  SYSTEM  REDUNDANCY,  and  RECOVERABILITY  REQUIREMENTS. 
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3.3.2. 1  Evaluation  Measure  for  Supportability 

Supportability  relates  to  a  system’s  ability  to  be  maintained  in  its  operational 
environment.  The  SV-7  must  be  altered  to  include  this  infonnation.  The  measure  for 
Supportability  is  simply  named  “SUPPORTABILITY  REQUIREMENTS.”  If  the  system  designers 
have  incorporated  supportability  requirements  into  the  design,  they  should  be  included  in  the  SV- 
7.  The  official  measure  simply  asks,  “Have  supportability  requirements  been  accounted  for?” 
The  term  Supportability  Requirement  refers  to  specific  systems  engineering  concepts.  Mean 
Time-Between-Maintenance  (MTBM)  and  Mean  Time-Between-Replacement  (MTBR)  measure 
the  time  that  the  system  is  active  between  scheduled  maintenance.  The  longer  this  time  is,  the 
more  maintenance  is  required;  therefore,  more  man-hours  are  required.  The  concepts  of  M 
(Mean  active  maintenance  time),  Mmax  (Maximum  maintenance  time),  and  MPT  (Mean  active- 
preventative-maintenance-time)  measure  how  much  time  is  actually  spent  in  the  maintenance  of 
the  system.  These  Supportability  concepts  basically  measure  the  “down-time”  and  time  between 
scheduled  maintenance.  At  this  point  of  design,  it  is  important  only  to  detennine  if  these  things 
have  been  considered  in  the  design  and  ensure  that  they  are  explicitly  stated  in  the  architecture. 

At  a  later  time,  it  will  be  important  to  ensure  that  the  times  are  acceptable  (although  the  system 
must  meet  the  basic  requirements  to  be  considered  a  viable  alternative).  Through  the  use  of 
these  “career-field  standard”  equations  (Blanchard  &  Fabrycky,  2006),  the  concept  of 
Supportability  may  be  measured,  as  these  are  the  ideas  that  are  intended  to  be  measured.  It  can 
be  assumed  that  a  system  is  supportable  if  it  exhibits  these  qualities.  The  question  of 
Supportability  is  a  constructed  idea  for  this  evaluation,  thus,  the  measure  is  Constructed-Proxy. 
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3.3.2.2  Evaluation  Measure  for  Reliability 

The  Reliability  value  has  characteristics  similar  to  Supportability.  Thought  it  measures  a 
separate  value,  the  way  in  which  they  are  measured  is  similar.  The  measure  “RELIABILITY 
REQUIREMENTS”  specifically  asks,  “Have  reliability  requirements  been  accounted  for?”  It  also 
looks  in  the  SV-7  for  similar  equations.  Reliability  seeks  to  measure  the  “up-time”  of  the  system 
or  the  length  of  time  that  it  is  operational  prior  to  disruption.  This  measure  assumes  that  proper 
preventive  maintenance  has  been  accomplished.  The  evaluator  should  look  for  such  equations  as 
the  Availability  family  (achieved:  Aa,  inherent:  A;,  operational:  Ao)  and  MTBF  (Blanchard  & 
Fabrycky,  2006).  This  measure  is  also  a  Constructed-Proxy  measure  as  it  uses  the  same  type  of 
standard  equations  to  measure  the  concepts  involved  in  this  value. 

3.3.2.3  Evaluation  Measure  for  Survivability 

Survivability  departs  slightly  from  the  method  used  to  determine  the  other 
Maintainability  values.  Survivability  measures  how  susceptible  a  system  is  to  hostile  action  and 
how  that  action  will  affect  the  system.  Redundant  systems  tend  to  remain  operational  through 
more  hostile  actions  than  systems  with  no  redundancies.  SYSTEM  REDUNDANCY  allows  a  back¬ 
up  system  to  take  the  place  of  the  primary  in  the  case  of  failure  or  attack,  thereby  allowing  the 
system  to  remain  operational.  The  “SYSTEM  REDUNDANCY”  measure  detennines  the  degree  to 
which  systems  are  redundant.  The  OV-6,  Operational  Event  Trace  Description,  will  provide 
some  insight  as  to  whether  there  are  intentional  system  redundancies  present.  Through  the 
OV-6’s  use  of  chronological  event  depiction,  it  is  possible  to  see  which  systems  are  performing 
similar  actions  simultaneously  and  whether  back-up  systems  exist  or  not.  The  measure  in  this 
case  is  Constructed-Proxy.  There  is  no  standard  way  to  measure  redundancy;  however,  looking 
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in  the  OV-6,  the  scorer  may  be  able  to  get  an  impression  as  to  whether  a  system  will  remain 
operational  after  attack. 

3.3.2 A  Evaluation  Measure  for  Recoverability 

Recoverability  returns  to  the  construct  in  place  for  the  Supportability  and  Reliability 
values.  Recoverability,  however,  measures  “RECOVERABILITY  REQUIREMENTS”  through  the  use 
of  Mean-Time-to-Repair  (MTTR)  and  MCT  (Mean  active-corrective-maintenance-time).  These 
ideas  measure  the  length  of  time  that  it  takes  to  actually  repair  the  system  after  some  failure. 
Whether  the  failure  is  through  enemy  attack  or  through  some  other  type  of  system  failure,  these 
equations  will  account  for  how  long  a  component  takes  to  repair.  The  SV-7  is  also  the  source  for 
these  concepts.  Recoverability  is  also  a  Constructed-Proxy  measure,  as  it  uses  standard 
equations  and  directly  measures  the  concept  of  recoverability. 

3.3.3  Interoperability  Measures 

The  final  branch  of  System  Effectiveness  determines  how  well  the  system  operates  with 
other  systems  and  between  its  own  nodes.  Interoperability  was  one  of  the  major  focuses  for  the 
sponsors  of  the  JFPASS;  therefore,  its  measurement  was  important  to  the  value  of  the 
architecture.  The  two  Interoperability  values  are  measured  through  the  use  of  three  total 
evaluation  measures. 

3.3.3. 1  Evaluation  Measure  for  Interchangeability 

Interchangeability  deals  with  the  ability  of  the  system  components  and  nodes  to  be 
interchanged  between  service  components.  In  the  Joint  environment,  being  able  to  operate 
smoothly  in  any  service  context  is  vital.  The  measure  “JOINT  OPERATIONS”  is  the  sole 
Interchangeability  measure,  and  it  determines  how  many  services  have  been  involved  in  the 
development  of  the  system.  The  existing  CONOPs  for  each  service  must  be  considered  so  that 
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no  critical  functions  are  ignored,  thereby  ensuring  a  fully  interoperable  system.  The  matter  of 
ensuring  that  each  service  component  can  operate  within  the  context  of  the  new  system  is  a 
matter  of  training,  but  ensuring  that  their  existing  CONOPs  and  critical  operations  have  been 
accounted  for  will  speed  the  process.  The  AV-1  document  will  outline  how  other  services  were 
incorporated  in  the  design  process,  but  the  OV-2,  OV-3,  and  OV-4  may  also  contain  infonnation 
regarding  the  other  services’  CONOPs  which  were  incorporated.  This  measure  is  a  Natural- 
Proxy  type.  The  number  of  services  is  a  constructed  way  of  measuring  this  concept,  and  its  way 
of  measuring  Interoperability  is  only  a  proxy  for  how  the  system  will  actually  allow  the 
Interchangeability  of  different  services. 

3.3.3.2  Evaluation  Measures  for  Communication 

The  ability  of  the  system  to  communicate  is  essential  to  its  basic  operation.  Since  this  is 
a  largely  automated  system,  the  nodes  must  communicate  with  one  another.  In  addition,  the 
system  must  be  able  to  communicate  its  processes  with  the  users.  The  Net-Centric  Enterprise 
Solutions  for  Interoperability  (NESI)  guidance  was  used  to  detennine  the  system’s  ability  to 
communicate.  The  two  Communication  measures  are  NESI  CONSIDERATION  and  NESI 
EVALUATION. 

3.3.3.2.1  NESI  CONSIDERATION 

“Was  NESI  guidance  taken  into  account  when  constructing  the  architecture?”  The 
system  designers  should  make  use  of  this  detailed  guidance  for  net-centric  systems.  Since  the 
JFPASS  is  required  to  be  net-centric  and  with  the  DoD’s  shift  to  net-centric  warfare,  some 
standard  of  net-centricity  is  required.  Currently,  NESI  is  the  only  set  of  consolidated  guidance 
for  determining  how  a  net-centric  system  should  look.  At  this  early  stage  of  development,  the 
consideration  of  the  NESI  guidance  in  the  design  is  the  best  way  to  measure  the  design 
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component  of  Communication.  At  a  later  time,  it  will  be  possible  to  measure  how  net-centric  a 
system  is,  but  components  must  be  in  operation  in  the  field  to  actually  see  how  they  will 
communicate.  The  primary  source  for  NESI  DEVELOPMENT  is  the  TV-1.  If  the  NESI 
documentation  is  not  listed  in  the  TV-1,  it  may  not  have  been  considered.  AV-1  is  a  possible 
back-up  as  well.  NESI  DEVELOPMENT  is  a  Constructed-Direct  measurement.  NESI  is  becoming 
the  DoD  standard  for  net-centric  designs  and  if  a  system  is  constructed  to  the  specifications 
contained  within,  then  it  is  generally  a  communicable  system. 

3.3.3.2.2  NESI  EVALUATION 

The  NESI  documentation  includes  many  evaluation  measures  of  its  own.  This  allows  the 
system  developers  to  perform  an  initial  evaluation  to  ensure  that  their  system  is  net-centric. 
Through  the  use  of  the  checklists  included  in  NESI,  the  NESI  EVALUATION  measure  can 
determine  how  the  system  developers  have  done.  Again,  at  this  stage  of  development,  it  is 
important  only  that  the  evaluation  has  been  completed.  The  actual  results  of  the  evaluation 
should  be  included  as  an  appendix  to  the  architecture,  but  the  quality  of  the  results  will  not  have 
as  heavy  of  an  influence  until  the  system  reaches  a  Milestone  B  approval  authority.  The  NESI 
EVALUATION  may  either  be  found  in  the  TV-1  or  as  an  appendix  to  the  architecture.  This  is  a 
Constructed-Proxy  measure.  Although  NESI  is  becoming  a  standard  tool,  the  evaluation  is  not  a 
direct  measure  for  this  model.  It  is  a  way  to  determine  if  the  evaluation  was  done  through 
another  system. 
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3.4  Create  Value  Functions 


Following  the  creation  of  the  full  hierarchy  including  values  and  measures,  Single 
Dimension  Value  Functions  (SDVFs)  were  assigned  to  each  measure.  The  SDVFs  served  to 
convert  the  measure’s  score  to  a  value,  based  on  the  range  of  the  measure.  These  SDVFs 
converted  the  individual  scales  to  value  units  ranging  from  zero  to  one.  These  value  scores  may 
then  be  summed  using  the  general  additive  value  function.  All  SDVFs  were  developed  in 
coordination  with  the  decision-maker. 

3.4.1  Capability  Measures  Functions 

The  OPERATIONAL  NEEDS  measure,  under  the  Purposefulness  value  was  scored  on  a  scale 
of  0  to  1 .  These  scores  represent  the  percentage  of  OPERATIONAL  NEEDS  addressed  by  functions. 
The  SDVF  is  a  monotonically  increasing  type;  therefore,  as  more  of  the  OPERATIONAL  NEEDS  of 
the  system  are  met,  more  value  is  gained.  The  value  is  gained  in  an  exponentially  increasing 
fashion,  so  that  the  difference  between  0.1  and  0.2  is  the  same  as  the  difference  between  0.7  and 
0.8.  Figure  3.6  displays  the  SDVF  for  OPERATIONAL  NEEDS.  This  SDVF  was  validated  by  the 
decision-maker  on  12  February  2009. 
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Figure  3.6.  OPERATIONAL  NEEDS  Single  Dimension  Value  Function 


83 


The  second  measure  of  Purposefulness,  THREAT  DETECTION,  is  a  binary  measure.  The 
acceptable  range  of  scores  for  THREAT  DETECTION  is  either  “no”  or  “yes.”  This  measure 
determines  if  a  Threat  Detection  plan  exists.  The  SDVF  for  THREAT  DETECTION  is  discrete,  with 
two  bins.  All  possible  value  is  earned  if  the  score  is  “yes”  and  no  value  is  earned  if  the  score  is 
“no.”  Figure  3.7  shows  a  generic  binary  SDVF,  which  may  be  applied  to  any  binary  measure. 

All  subsequent  binary  measures,  of  which  there  are  a  total  of  nine,  use  the  SDVF  in  Figure  3.7. 
The  SDVF  was  validated  by  the  decision-maker  on  20  November  2008. 
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Figure  3.7.  Generic  Binary  Single  Dimension  Value  Function 


The  last  two  measures  of  Purposefulness,  THREAT  ASSESSMENT  and  WARNING  PLAN,  are 
also  binary  measures.  These  measures  determine  if  either  a  Threat  Assessment  or  Warning  Plan 
exists.  The  SDVF  for  both  is  discrete,  binary  as  well.  All  possible  value  is  earned  if  the  score  is 
“yes”  and  no  value  is  earned  if  the  score  is  “no.”  Figure  3.7  shows  the  binary  SDVF,  which  was 
validated  for  this  measure  by  the  decision-maker  on  20  November  2008. 
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The  first  measure  of  Practicality,  TECHNOLOGICAL  AVAILABILITY,  is  based  on  a  widely 
used  scale  called  Technology  Readiness  Levels.  This  is  a  9-level  scale;  therefore,  the  SDVF  is 
discrete  with  nine  bins.  TECHNOLOGICAL  AVAILABILITY  determines  the  level  at  which 
technology  is  available  for  this  project.  Table  3.5  shows  all  TRLs  with  their  definitions.  As 
TRLs  increase,  more  value  is  gained.  Though  this  Value  Function  is  discrete,  the  bins  take  on  an 
exponentially  increasing  value  curve.  Therefore,  the  difference  between  TRL  1  and  TRL  2  is 
much  smaller  than  the  difference  between  TRL  7  and  TRL  8.  Figure  3.8  demonstrates  this 
concept.  The  SDVF  was  validated  by  the  decision-maker  on  1 1  January  2009. 


Table  3.5.  Technology  Readiness  Levels 


TRL  1 

Basic  Principles  observed  and  reported  -  Lowest  level  of  technology  readiness 

TRL  2 

Technology  concept  and/or  application  formulated  invention  begins 

TRL  3 

Analytical  and  experimental  critical  function  and/or  characteristic  proof  of  concept 

TRL  4 

Component  and/or  breadboard  validation  in  laboratory  environment 

TRL  5 

Component  and/or  breadboard  validation  in  relevant  environment 

TRL  6 

System/sub  system  model  or  prototype  demonstration  in  a  relevant  environment 

TRL  7 

System  prototype  demonstration  in  an  operational  environment 

TRL  8 

Actual  system  completed  and  ‘flight  qualified’  through  test  and  demonstration 

TRL  9 

Actual  system  ‘flight  proven’  through  successful  mission  operations 

Figure  3.8.  TECHNOLOGICAL  AVAILABILITY  Single  Dimension  Value  Function 
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The  second  measure  of  Practicality  is  ENVIRONMENTAL  IMPACT.  ENVIRONMENTAL 
IMPACT  is  a  discrete  value  function  with  four  bins.  It  is  intended  to  measure  the  level  of  impact 
that  a  given  system  has  on  its  environment.  For  this  measure,  the  bins  were  designed  to  capture 
increasing  stringency  of  environmental  laws  and  restrictions.  In  this  case,  the  lowest  value  is 
that  a  system  cannot  be  built  within  any  environmental  restrictions,  i.e.,  it  will  have  a  vast 
detrimental  effect  on  the  environment.  Contingency  Operations  environmental  constraints  are 
next  in  level  of  restrictiveness.  Following  that,  the  CONUS  or  Contingency  constraints  provide  a 
higher  level  of  restriction.  Finally,  since  a  system  would  have  to  comply  with  three  separate 
levels  of  restriction;  CONUS,  Contingency,  or  Host  Nation  constraints  is  the  most  restrictive 
level  of  ENVIRONMENTAL  IMPACT.  It  is  assumed  that  it  is  easier  to  design  for  CONUS 
constraints  than  it  is  to  design  for  Host  Nation  constraints,  since  the  corporations  and  designers 
are  familiar  with  the  CONUS  laws.  There  is  a  jump  in  value  between  having  a  system  that  can 
be  built  in  Contingency  and  CONUS  constraints.  Figure  3.9  displays  the  SDVF  for 
ENVIRONMENTAL  IMPACT.  This  SDVF  was  validated  on  20  November  2009. 


Figure  3.9.  ENVIRONMENTAL  IMPACT  Single  Dimension  Value  Function 
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The  third  measure  of  Practicality  is  MONETARY  PRACTICALITY  -  INITIAL.  This  measure 
is  a  discrete  SDVF  with  three  bins.  It  is  intended  to  measure  the  attainment  of  the  ability  to 
construct  a  system  within  budgetary  constraints.  To  this  end,  the  three  bins  were  constructed  to 
capture  situations  in  which  the  system  estimates  fall  above,  within,  and  below  budget.  The 
“Within  Budget”  bin  refers  specifically  to  estimates  which  fall  within  +1-5%  of  the  given 
program  budget.  Therefore,  any  estimate  falling  above  5%  of  budget  qualifies  as  “Above 
Budget,”  while  any  estimate  falling  below  5%  of  budget  is  considered  to  “Save  Money.”  Figure 
3.10  displays  the  SDVF  for  this  MONETARY  PRACTICALITY  measures.  It  was  validated  by  the 
decision-maker  on  12  February  2009. 
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Figure  3.10.  MONETARY  PRACTICALITY  Single  Dimension  Value  Function 
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The  last  measure  of  Practicality,  MONETARY  PRACTICALITY  -  MAINTENANCE,  is  very 
similar  to  the  previous  measure,  except  that  it  refers  to  the  life-cycle  cost  of  the  project.  The 
SDVF  converts  value  in  the  same  way,  with  the  same  levels.  This  is  demonstrated  in  Figure 
3.10.  This  SDVF  was  also  validated  on  12  February  2009. 

ADAPTION  is  the  single  measure  of  Flexibility.  ADAPTION  measures  the  degree  to  which 
the  system  is  able  to  adapt  to  changing  operational  requirements.  This  is  measured  on  a  discrete 
scale  with  five  bins.  Each  bin  measures  an  increasingly  easier  method  of  changing  the  system 
configuration.  The  lowest  bins  and  therefore  of  lowest  value  is  “Static,”  meaning  that  the  system 
is  not  capable  of  being  changed  once  it  is  implemented.  The  next  level  is  “Unacceptable  Effort,” 
which  refers  to  a  system  which  can  be  changed,  but  is  cost  and/or  time  restrictive  to  actually 
make  the  change  to  meet  the  mission  at  hand.  “3rd  Party  Acceptable  Effort”  refers  to  a  situation 
in  which  the  system  can  be  changed  within  cost  and  time  constraints,  but  a  3ld  party  must  be 
“imported”  to  make  the  change.  In  this  case,  users  are  not  capable  of  changing  the  system  as 
needed.  The  next  level  is  “On-Site  Acceptable  effort.”  In  this  case,  the  users  are  capable  of 
making  the  change  within  cost  and  time  constraints,  though  it  may  require  such  considerations 
and  consultation  from  system  designers,  system  downtime,  or  added  cost.  The  final  bin  and  of 
most  value  to  the  decision-maker  is  “Minimal  Effort.”  This  refers  to  a  system  that  is  flexible  by 
its  very  nature.  Any  changes  to  meet  operations  requirements  are  quickly  and  easily  made  with 
little  to  no  additional  time  or  cost.  As  the  system  gets  easier  to  change,  more  value  is  added, 
with  a  significant  jump  in  value,  0.4  value  units,  between  “Unacceptable  Effort”  and  “3rd  Party 
Acceptable  Effort.”  This  value  jump  represents  the  value  to  the  decision-maker  of  having  a 
system  that  is  capable  of  being  changed.  Figure  3.11  shows  the  SDVF  for  ADAPTION,  which  was 
validated  by  the  decision-maker  on  20  November  2008. 
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Figure  3.11.  ADAPTION  Single  Dimension  Value  Function 


3.4.2  Maintainability  Measures  Value  Functions 

SUPPORT  ABILITY  REQUIREMENTS  is  the  first  of  the  Maintainability  measures. 
Specifically,  SUPPORT  ABILITY  REQUIREMENTS  measures  the  Supportability  value  by 
determining  if  the  system  designers  have  considered  supportability  issues.  This  is  a  discrete, 
binary  SDVF;  with  its  range  being  either  “no”  or  “yes.”  If  the  equations  related  to  supportability 
were  mentioned,  then  the  system  gains  full  credit  (“yes”);  if  these  equations  were  not  considered, 
the  system  earns  a  “no.”  If  the  SV-7  product  does  not  exist,  the  system  will  also  score  “no,” 
since  the  SV-7  is  required  to  determine  the  design  tolerances  used  for  the  component  systems. 
Figure  3.7  shows  the  SDVF  for  SUPPORTABILITY  REQUIREMENTS,  which  was  validated  on  20 
November  2008. 

The  RELIABILITY  REQUIREMENTS  measure  is  similar  to  Supportability,  in  that  it  is  also 
binary  and  seeks  to  measure  the  use  of  reliability  style  equations  in  the  design  of  the  system. 
RELIABILITY  REQUIREMENTS  is  the  second  of  the  Maintainability  values  and  specifically 
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measures  the  Reliability  value.  Reliability  considerations  are  a  different  set  of  design  criteria,  but 
are  also  found  in  the  SV-7.  Value  is  earned  in  the  same  way  as  SUPPORT  ABILITY 
REQUIREMENTS.  Figure  3.7  demonstrates  this  in  the  SDVF.  This  SDVF  was  validated  on  20 
November  2008. 

SYSTEM  REDUNDANCY  measures  the  degree  of  attainment  of  the  Survivability  value.  If  a 
system  has  redundancies,  then  it  is  more  likely  to  survive  an  attack.  To  accomplish  this 
measurement,  a  discrete  value  function  with  four  bins  was  created.  As  the  redundancies  on 
systems  increase,  more  value  is  earned.  The  bins  represent  a  value  “jump”  when  multiple 
redundancies  are  considered  for  systems.  The  lowest  bin  is  “No  redundancy,”  meaning  that  all 
systems  are  stand-alone  and  would  constitute  a  loss  of  mission  effectiveness  if  they  were 
destroyed.  The  next  bin,  “Some  Systems  have  Single  Redundancies,”  captures  the  idea  that 
some  systems  are  given  a  single  back-up  to  ensure  their  operation.  The  decision-maker  felt  that 
it  was  important  for  systems  to  have  more  than  a  single  redundancy  in  a  force  protection 
scenario;  therefore,  the  next  bin,  “Some  Systems  have  Multiple  Redundancies,”  captures  the  next 
level,  which  adds  a  great  deal  of  value  to  the  measure.  Finally,  “All  Systems  have  Multiple 
Redundancies”  is  the  highest  bin  and  level  of  value.  Figure  3.12  displays  the  SDVF  for  SYSTEM 
REDUNDANCY.  This  SDVF  was  validated  on  20  November  2008. 
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Figure  3. 12.  SYSTEM  REDUNDANCY  Single  Dimension  Value  Function 


The  RECOVERABILITY  REQUIREMENTS  measure  is  the  last  of  the  Maintainability 
measures,  which  specifically  measures  the  value  of  Recoverability.  This  measure’s  SDVF 
follows  the  same  example  as  both  SUPPORT  ABILITY  REQUIREMENTS  and  RELIABILITY 
REQUIREMENTS.  It  is  a  discrete,  binary  SDVF.  In  this  case,  the  equations  generally  associated 
with  recoverability  are  looked  for  in  the  SV-7  product.  Figure  3.7  shows  the  SDVF  for 
RECOVERABILITY  REQUIREMENTS.  It  was  validated  on  20  November  2008. 

3.4.3  Interoperability  Measures  Value  Functions 

JOINT  OPERATIONS  is  the  single  measure  associated  with  the  Interchangeability  value. 
This  measure  determines  whether  multiple  service  components’  CONOPs  have  been  considered 
in  the  design  of  the  system.  This  is  a  binary,  discrete  SDVF.  In  order  to  receive  value  for  this 
measure,  the  system  must  incorporate  all  services’  CONOPs.  No  credit  is  given  unless  all  four 
service  components  have  been  considered.  The  SDVF  was  constructed  in  this  manner  to  ensure 
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that  the  importance  of  having  all  service  components  is  captured.  Without  all  four  services,  the 
interchangeability  of  the  system  is  of  no  value.  The  range  for  this  SDVF  is  also  “yes”  or  “no.” 
The  only  way  to  score  100%  value  or  a  “yes,”  is  to  have  all  service  components’  CONOPs 
considered  in  the  design.  Figure  3.7  displays  the  validated  (on  20  November  2008)  SDVF. 

NESI  DEVELOPMENT  is  the  measure  created  to  detennine  the  degree  of  attainment  of  the 
Communication  Value.  This  measure  seeks  to  detennine  if  the  Net-Centric  Enterprise  Solutions 
for  Inoperability  guidance  has  been  taken  into  account  for  the  design  of  the  system.  This  is  a 
binary,  discrete  SDVF,  measuring  either  “yes”  or  “no”  as  to  whether  NESI  has  been  used. 

Figure  3.7  shows  the  binary  SDVF  used  for  NESI  DEVELOPMENT.  This  SDVF  was  validated  on 
20  November  2008. 

NESI  EVALUATION  measures  the  same  value  in  the  same  way  as  NESI  DEVELOPMENT. 
NESI  EVALUATION,  however,  seeks  to  detennine  if  the  system  designers  have  completed  an 
evaluation  on  the  system.  Supplied  in  the  NESI  documentation  are  several  checklists  and 
measures  for  determining  the  net-centricity  of  a  system.  The  system  designers  must  have 
completed  these  evaluations  to  gain  credit  for  NESI  EVALUATION.  Figure  3.7  shows  the  SDVF 
for  NESI  EVALUATION.  This  SDVF  was  validated  on  20  November  2008. 

3.5  Value  Hierarchy  Weights 

To  detennine  the  importance  of  each  measure,  values  must  first  be  individually  weighted. 
By  detennining  local  weights,  the  global  weights  are  easily  found.  This  study  uses  primarily  the 
direct  weighting  procedure,  although  an  additional  procedure  using  “Tornado  Charts”  was  used 
for  validation  and  final  weight  determination.  Table  3.6  shows  each  System  Effectiveness  tier’s 
values  with  their  global  and  local  weights.  The  global  weights  for  at  each  tier  sum  to  0.6  to 


92 


represent  the  60%  of  the  fundamental  value  accounted  for  by  the  System  Effectiveness  branch. 
All  weights  in  sections  discussed  in  section  3.5  represent  the  final  weights  following  validation. 


Table  3.6.  Value  Weights 


Value 

Tier 

Local  Weight 

Global  Weight 

System  Effectiveness 

1 

0.6 

0.6 

Capability 

2 

0.45 

0.27 

Maintainability 

2 

0.275 

0.165 

Interoperability 

2 

0.275 

0.165 

Purposefulness 

3 

0.6 

0.162 

Practicality 

3 

0.3 

0.081 

Flexibility 

3 

0.1 

0.027 

Dependability 

3 

0.6 

0.099 

Resiliency 

3 

0.4 

0.066 

In  terchangeabi  lity 

3 

0.3 

0.05 

Communication 

3 

0.7 

0.116 

Supportability 

4 

0.35 

0.035 

Reliability 

4 

0.65 

0.064 

Survivability 

4 

0.6 

0.04 

Recoverability 

4 

0.4 

0.026 

3.5.1  Tier  1  Weights 

A  top-down  approach  was  used  for  the  majority  of  the  weighting,  although  validation  was 
completed  in  a  bottom-up  fashion.  Tier  one  was  the  first  tier  to  be  weighted  (top-down).  With 
the  split  between  System  Effectiveness  and  Architecture  Quality,  relative  weights  had  to  be 
determined  to  find  how  important  each  of  the  two  branches  would  be.  The  local  weights  were 
found  to  be  0.6  and  0.4,  respectively.  These  weights  account  for  the  importance  of  the 
instantiated  system  versus  the  architectural  products.  While  the  architectural  products  are  very 
important,  particularly  in  the  acquisition  realm,  the  value  of  the  system  being  represented  is  of 
more  importance. 
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3.5.2  Tier  2  Weights 

The  three  values  considered  on  Tier  2  of  the  Hierarchy  were  Capability,  Maintainability, 
and  Interoperability.  A  direct  weighting  scheme  (“100  coin  method”)  was  used  for  these  values. 
Initially,  Capability  was  weighted  at  0.40,  with  Maintainability  and  Interoperability  both  being 
0.30  of  the  value  of  System  Effectiveness.  The  decision-maker  agreed  that  Maintainability  and 
Interoperability  were  in  fact  of  equal  weight,  but  the  importance  of  Main  tainability  and 
Interoperability  in  relation  to  Capability  was  adjusted.  The  final  value  of  Capability  was  found 
to  be  0.45  of  System  Effectiveness ,  while  Maintainability  and  Interoperability  were  0.275  each. 
These  final  values  incorporate  the  changes  made  during  weighting  validation.  Capability  has  the 
highest  weighting  due  to  the  importance  that  the  system  is  capable  of  perfonning  its  intended 
operations.  If  a  system  cannot  do  what  it  is  intended  to  do,  it  is  little  value  to  the  user;  therefore, 
the  0.45  weighting  accounts  for  this  major  importance  in  terms  of  the  system’s  ability  to  do  its 
job.  Maintainability  and  Interoperability  are  both  important  to  the  decision-maker,  but  are 
overshadowed  by  Capability. 

3.5.3  Tier  3  Weights 

Tier  three  was  weighted  next  using  the  local  weighting  method.  These  values  were 
examined  branch-by-branch  to  ensure  that  each  value  was  accounted  for  and  no  areas  were  left 
unconsidered.  Following  the  weighting,  each  value  on  Tier  3  was  validated. 

3.5.3. 1  Tier  3  Capability  Branch  Weights 

The  three  values  under  Capability  are  Purposefulness,  Practicality,  and  Flexibility.  It 
was  unanimously  agreed  among  the  decision-maker  and  subject  matter  experts  that 
Purposefulness  was  by  far  the  most  important  value  -  possibly  in  the  entire  hierarchy.  With  that 
knowledge,  Practicality  and  Flexibility  were  weighted  using  a  swing-weight  style  approach. 
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Practicality  was  agreed  to  be  less  important  than  Purposefulness  and  Flexibility  less  important 
than  Practicality.  Next,  relative  weights  were  examined.  It  was  detennined  that  Practicality  is 
approximately  three  times  as  important  as  Flexibility.  This  decision  was  made,  because  if  a 
system  cannot  be  practically  constructed,  then  its  flexibility  does  not  matter.  Purposefulness  was 
then  agreed  to  be  twice  as  important  as  Practicality.  This  decision  was  made  because  a  system 
may  be  practical,  but  if  it  does  not  accomplish  its  goals,  there  is  no  need  to  construct  the  system 
in  the  first  place.  This  process  yielded  weights  of  0.60  for  Purposefulness,  0.30  for  Practicality, 
and  0.10  fox  Flexibility. 

3.5.3.2  Tier  3  Maintainability  Branch  Weights 

The  Maintainability  branch  is  the  only  branch  within  System  Effectiveness  that  contains  a 
fourth  tier  of  decomposition.  The  first  step  for  weighting  this  branch  was  to  determine  the 
weights  of  the  tier-three  values.  The  tier- four  values  were  examined  after  all  tier-three  values 
were  weighted.  Maintainability  has  sub-values  of  Dependability  and  Resiliency.  Dependability 
refers  to  a  system’s  maintainability  during  peace-time  operations,  and  Resiliency  refers  to  its 
maintainability  during  hostile  actions.  The  weighting  for  these  values  was  based  on  the 
frequency  of  occurrence.  Since  the  U.S.  military  has  more  assets  that  are  operating  in  peace-time 
operations,  Dependability  was  determined  to  be  more  important.  While  the  military  has  more 
operations  engaged  in  peaceful  actions,  the  operations  vulnerable  to  hostile  actions  are 
considered  to  be  of  more  importance  to  the  completion  of  the  National  Security  and  National 
Military  strategies.  This  idea  was  confirmed  by  Subject  Matter  Experts.  Therefore,  Resiliency 
was  weighted  at  0.40  and  Dependability  was  weighted  at  0.60.  Dependability  being  only  slightly 
more  important  accounts  for  the  mission  impact  of  contingency  operations,  but  the  overall 
importance  of  the  military’s  peace-time  operations. 
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3.5.3.3  Tier  3  Interoperability  Branch  Weights 

The  Interoperability  branch  was  the  last  branch  of  System  Effectiveness  to  be  weighted. 
This  branch  accounts  for  Interchangeability  and  Communication.  The  swing  weighting  method 
was  used  for  this  weighting.  It  was  detennined  first  the  Communication  was  more  important  to 
the  decision-maker;  meaning  that  the  nodes  communicating  effectively  is  more  important  than 
the  components  having  the  ability  to  be  interchanged.  With  Interchangeability  being  the  less 
important  value,  the  discussion  led  to  a  determination  that  Communication  was  four  times  more 
important  than  Interchangeability.  This  order  of  magnitude  captures  the  extreme  importance  of 
the  initial  nodes  perfonning  their  communication  function  first.  Their  Interchangeability  falls 
behind,  since  without  communication,  there  would  be  no  need  to  interchange. 

3.5.4  Tier  4  Weights 

Following  the  completion  of  tier-three  weighting,  tier-four  weights  were  examined.  Only 
three  values  of  the  hierarchy  have  tier-four  values.  These  values  were  examined  individually  to 
find  their  local  weights.  Under  the  Maintainability  branch,  the  values  of  Dependability  and 
Resiliency  were  decomposed  by  one  additional  level,  giving  four  tier-four  values  under 
Maintainability . 

Dependability  was  examined  first.  The  sub-values  of  Supportability  and  Reliability 
compose  the  idea  of  peace-time  dependability.  Supportability  represents  the  ability  to  maintain 
the  system  and  Reliability  represents  the  system’s  ability  to  continue  standard  operations  if 
maintained  properly.  Supportability  was  found  to  be  less  important  than  Reliability,  based  on  the 
need  for  system  “up-time.”  If  a  system  is  difficult  to  maintain,  the  system  will  be  offline  more 
often.  The  most  important  thing  about  standard  peace-time  operations  is  that  the  system  is 
operational  more  often  than  it  is  not.  Therefore,  the  decision-maker  was  willing  to  sacrifice 
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maintainability  for  more  operational  time.  Initially,  Reliability  was  found  to  be  twice  as 
important  as  Supportability,  but  the  validation  phase  found  that  Reliability  was  actually  0.65  of 
the  value  of  Dependability  and  Supportability  was  0.35  of  the  value.  This  determination  was 
made  using  the  “100  coin  method”  during  validation. 

The  second  aspect  of  Maintainability,  Resiliency,  was  weighted  next.  A  similar 
consideration  was  made  for  this  value.  It  is  of  more  importance  that  the  system  remains 
operational  during  an  attack  than  it  is  easily  repaired  after  the  attack.  Generally,  more  time  is 
available  following  an  attack  and  repair  is  not  as  critical;  therefore,  ensuring  that  the  system 
never  goes  down  is  important.  The  direct  weighting  method  was  used  for  the  values  of 
Survivability  and  Recoverability.  Survivability  was  found  to  be  0.60  of  the  value  of  Resiliency 
and  Recoverability  0.40. 

3.5.5  Weight  Validation 

Following  the  initial  value  weighting,  the  entire  Hierarchy  was  reviewed  to  ensure 
weights  were  accurate.  The  weights  were  validated  using  the  “tornado  chart”  weighting  method. 
Each  level  of  the  hierarchy  was  examined  from  a  “bottom-up”  method.  The  bottom-up  method 
was  chosen,  so  that  as  measures  were  re-weighted  and  adjusted,  the  weights  of  the  higher  levels 
could  immediately  reflect  the  changes.  A  major  advantage  of  the  tornado  weighting  method  is 
that  it  allows  the  decision-maker  to  see  the  relative  importance  of  values  in  different  branches 
and  ensure  that  their  relative  placement  is  correct;  therefore,  global  weights  were  used  during  the 
validation  process.  Global  weights  are  the  weights  of  the  value  relative  to  all  other  values  on  the 
same  tier  of  the  hierarchy. 
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3.5.5.1  Tier  3  Validation 


Tier  three  was  the  first  level  to  be  validated  using  a  “stacked”  tornado  chart.  Since  there 
is  not  a  full  fourth  tier,  it  was  not  possible  to  start  on  the  fourth  level.  By  stacking  Tier  4  values 
and  displaying  on  a  single  bar  chart,  the  decision-maker  was  able  to  see  both  the  tier-three  values 
and  the  tier-four  values  at  the  same  time.  Weighting  was  first  done  using  separate  Tier  1 
branches  ( System  Effectiveness  and  Architecture  Quality)  and  then  combined  into  a  single  chart. 
In  each  chart,  the  relative  placement  of  each  value  was  individually  examined  to  ensure  its 
placement  in  the  overall  hierarchy.  Figure  3.13  shows  the  tier-three  global  weights  for  each 
value  in  the  system  branch.  When  local  weights  are  multiplied  up  through  the  hierarchy  to  be 
converted  to  global  weights,  their  order  of  importance  is  found. 


Purposefulness 

Communication 

Dependability 
Practicality 
Resilliancy 
Interchangeability 
Flexibility 

0.000  0.020  0.040  0.060  0.080  0.100  0.120  0.140  0.160  0.180 

Figure  3.13.  Tier  3  System  Effectiveness  Global  Weights  Stacked 
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In  the  initial  presentation  of  the  value  weights,  Purposefulness  was  weighted  similarly, 
but  shared  the  Capability  value  with  only  one  other  value.  Once  Flexibility  was  moved  to  the 
Capability  branch,  each  of  the  three  values  under  Capability  were  re -weighted.  Purposefulness 
maintained  its  original  weight  of  0.60  locally,  giving  it  a  global  value  of  0. 162.  It  was 
unanimously  agreed  that  Purposefulness  should  be  the  most  important  value  globally. 
Practicality  was  discussed  next.  Its  original  weight  was  0.40,  but  with  the  addition  of  Flexibility, 
was  re-weighted  to  be  0.30  of  the  value  of  Capability  and  0.081  global  value.  Practicality  was 
important  enough  to  be  in  the  top  of  values  of  importance,  therefore  its  weight  was  monitored  to 
ensure  that  it  remained  within  the  top  five  values.  Its  final  global  importance  was  fifth  overall. 
Flexibility  was  weighted  at  0. 10  as  discussed  in  section  3.5.3. 1 .  Its  global  value  was  0.027. 
Flexibility's  final  importance  was  fourteenth  of  importance  globally. 

The  tier  three  Maintainability  values  were  examined  next.  Dependability  and  Resiliency, 
with  their  sub  values  of  Reliability,  Supportability,  Survivability,  and  Recoverability  were 
validated  by  adjusting  the  weight  of  the  subvalues  to  determine  what  effect  it  had  on  the  overall 
values  of  Dependability  and  Resiliency.  Reliability  and  Supportability,  the  Dependability 
subvalues  on  tier  four,  were  originally  weighted  at  0.60  and  0.40  locally.  These  weights  were 
adjusted  to  0.65  for  Reliability  and  0.45  for  Supportability.  This  change  was  made  to  account  for 
the  fact  that  Reliability,  being  the  system’s  “up-time”  is  more  important  than  the  ease  of 
maintenance  for  a  system.  The  5%  change  to  each  value  was  made  to  adjust  their  standings  in 
the  tornado  chart.  When  examined  together,  these  values  make  up  100%  of  the  Dependability 
value.  Their  global  values  were  equal  to  0.064  for  Reliability  and  0.035  for  Supportability. 

These  global  values  were  used  to  examine  the  value’s  standings  on  the  Tornado  chart.  This 
change  placed  Dependability  above  Understandability  in  the  global  rankings  Tornado  chart 
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(Figure  3.46)  and  in  the  top  five  of  all  values.  Dependability  is  more  important  than 
understandability,  because  the  system’s  ability  to  continue  perfonnance  (maintenance  time  and 
system  up-time)  is  more  important  to  the  decision-maker  than  the  ease  of  reading  architectural 
products.  It  also  assured  Dependability' s  place  above  Practicality  on  the  System  Effectiveness 
values  only  tornado  chart  (Figure  3.44).  Dependability  itself  ( Reliability  plus  Supportability)  is 
weighted  as  0.60  local  and  0.099  global  on  tier  three.  Dependability  is  the  third  most  important 
value  globally. 

Resiliency  is  composed  of  the  subvalues  Survivability  and  Recoverability.  Survivability 
and  Recoverability  were  both  accepted  by  the  decision-maker  with  the  weights  as  presented.  The 
Survivability  value  was  presented  as  0.60  local  and  0.04  global.  Recoverability  was  0.40  local 
and  0.026  global.  These  weights  were  assigned  as  such  to  account  for  the  importance  of  the 
system  remaining  operational  during  an  attack.  It  is  important  for  the  system  to  be  recovered 
quickly,  but  the  more  critical  time  period  is  during  the  attack  itself.  These  subvalues  combine  to 
fonn  the  tier  three  value  of  Resiliency.  Resiliency,  on  tier  three  is  weighted  as  0.40  local  and 
0.066  global.  These  weightings  place  Resiliency  as  the  sixth  most  important  value  globally. 

Finally,  the  Interoperability  branch  was  examined  on  tier  three.  The  two  tier  three 
Interoperability  values  were  Interchangeability  and  Communication.  Due  to  the  movement  of 
Flexibility  from  the  Interoperability  branch  to  the  Capability  branch,  Communication  was  re¬ 
weighted  to  0.70  from  0.60  locally.  This  left  Communication  at  0. 1 16  global  value,  making  it 
the  second  most  important  value  globally.  Interchangeability  remained  at  0.30  local  weight.  Its 
global  value  was  set  at  0.05  and  was  the  seventh  most  important  global  value. 

Figure  3.14  shows  all  global  values  on  tier  three  of  the  hierarchy.  This  Tornado  chart 
includes  both  System  Effectiveness  values  and  Architecture  Quality  values.  As  shown  on  this 
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chart,  six  of  the  seven  most  important  values  are  from  the  System  Effectiveness  branch  of  the 
hierarchy.  These  rankings  were  validated  based  on  the  importance  of  the  system  to  perfonn  as 
expected.  The  Architectural  products  are  also  important,  but  they  only  represent  the  instantiated 
system  and  therefore  rank  lower  globally.  The  one  Architecture  Quality  value  in  the  top  seven  is 
Understandability.  This  value  is  placed  in  its  location  due  to  the  importance  of  the  architectural 
products  to  be  understood  and  to  effectively  communicate  the  concepts  of  the  future  system  to  a 
wide  audience. 


Purposefulness 
Communication 
Dependability 
Understandability 
Practicality 
Resilliancy 
Interchangeability 
Longevity 
Protectability 
Controllability 
Subscribability 
Compliancy 
SME  Input 
Flexibility 
Tailorability 
Scalability 
Traceability 
Consistency 
Evolvability 

0.000  0.020  0.040  0.060  0.080  0.100  0.120  0.140  0.160 


Figure  3.14.  Tier  3  All  Global  Values 
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3.5.5.2  Tier  2  Validation 


Tier  two  validation  was  done  using  the  same  method  as  tier  three.  Figure  3.46  shows  the 
final  global  and  local  weights  for  the  System  Effectiveness  values.  Following  the  weighting  of 
the  tier  three  values,  tier  two  values  were  simply  checked  for  their  placement  among  other 
System  Effectiveness  values  and  other  tier  2  values.  Figure  3.15  shows  all  values  in  Tier  2. 
Capability  was  validated  at  0.45  local  weight  and  0.27  global  weight.  Interoperability  and 
Maintainability  were  both  validated  at  0.275  local  and  0.165  global  weight. 


Figure  3.15.  Tier  2  System  Effectiveness  Local  and  Global  Weights 
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When  examined  among  other  tier  2  values,  Capability  was  still  the  most  important  value 
with  its  0.27  global  value.  Interoperability  and  Maintainability  were  the  next  most  important  tier 
two  values  at  0.165  global  weight  each.  This  ranking  confirmed  Capability  as  the  highest 
weighted  value. 

3.5.5.3  Tier  1  Validation 

The  final  validation  completed  was  for  the  tier  one  values.  The  only  tier  one  values  are 
the  two  major  branches  of  the  hierarchy.  System  Effectiveness  was  weighted  at  0.60  of  the 
fundamental  objective  and  Architecture  Quality  at  0.40  of  the  fundamental  objective.  This 
accounts  for  the  higher  importance  of  the  instantiated  system,  versus  the  architectural  products 
that  represent  the  system. 
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Chapter  4.  Analysis  and  Results 


Following  the  finalization  of  the  value  hierarchy  the  existing  architecture  was  scored  and 
future  alternatives  were  generated.  For  the  Value-Driven  Enterprise  Architecture,  the  JFPASS 
system  was  evaluated  first.  Following  this  baseline  score,  the  other  generated  alternatives  were 
scored  for  comparison  purposes.  A  sensitivity  analysis  was  then  completed  to  detennine  the 
effect  that  the  weights  of  each  value  had  on  the  final  score  as  well  as  the  impact  of  each  measure 
to  the  final  score. 

4,1  JFPASS  Architecture  Scoring 

The  JFPASS  system  was  examined  first.  Each  measure  in  the  hierarchy  was  examined 
individually  and  assigned  a  score  based  on  the  existing  architecture.  Since  the  existing 
architecture  is  at  a  very  early  stage  of  development,  there  are  portions  of  value  that  have  not  yet 
been  earned.  The  instantiated  system  does  not  yet  exist;  therefore,  only  the  architectural 
products  were  used  as  data  sources  in  the  scoring  process.  Some  measures  may  score  higher 
based  on  information  not  yet  included  in  architectural  products,  but  since  this  information  is  not 
readily  available  to  a  third  party  reviewer  and  is  therefore  not  verifiable,  it  cannot  be  included  in 
the  scoring  of  the  system.  Some  areas  of  value  are  located  in  very  specific  architectural 
products;  therefore,  if  these  products  do  not  yet  exist,  some  measures  cannot  be  accurately  scored 
until  those  products  are  created.  Value  will  be  earned  incrementally  as  new  information  is 
available  during  the  life  cycle  of  the  project.  The  scoring  presented  here  is  the  baseline  for  future 
improvements.  The  views  available  for  scoring  the  JFPASS  are  listed  in  Table  4. 1 .  Table  4.2 
shows  the  final  scores  for  each  of  the  measures.  To  score  the  alternatives,  the  hierarchy  was 
entered  into  a  proprietary  software  package,  Hierarchy  Builder  vl.01©  (Weir,  2006).  The 
Hierarchy  Builder  ©  package  allows  for  all  aspects  of  the  scoring  and  analysis  of  a  hierarchy. 
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Hierarchy  Builder  ©  also  generates  the  Single  Dimension  Value  Functions  (SDVF)  as  entered  by 
the  user.  To  determine  the  equations  associated  with  the  continuous  SDVFs,  a  trendline  was  fit 
to  the  curve  and  the  equation  was  exported  from  Microsoft  Excel©. 


Table  4.1  Available  JFPASS  views 


AV-1 

SV-1 

OV-1 

SV-2 

OV-2 

SV-4 

OV-4 

SV-6 

OV-5 

TV-1 

OV-6c 

Table  4.2  Scores  and  Associated  Values 


Alternative  Name 

JFPASS  17  February  2009 

Measure 

Score 

Global 

Weight 

Value 

OPERATIONAL  NEEDS 

0 

0.041 

0.001 

THREAT  DETECTION 

Yes 

0.041 

1.000 

THREAT  ASSESSMENT 

Yes 

0.041 

1.000 

WARNING  PLAN 

Yes 

0.041 

1.000 

TECHNOLOGICAL  AVAILABILITY 

TRL  1 

0.02 

0.050 

ENVIRONMENTAL  IMPACT 

CONU.S.  and  Contingency 
constraints 

0.02 

0.800 

MONETARY  PRACTICALITY  - 
INITIAL 

Cost  Unknown 

0.02 

0.000 

MONETARY  PRACTICALITY  - 
MAINTENANCE 

Cost  unknown 

0.02 

0.000 

ADAPTATION 

Static 

0.027 

0.000 

SUPPORT  ABILITY  REQUIREMENTS 

No 

0.035 

0.000 

RELIABILITY  REQUIREMENTS 

No 

0.064 

0.000 

REDUNDANCY 

Some  systems,  single  redundancy 

0.04 

0.250 

RECOVERABILITY 

REQUIREMENTS 

No 

0.026 

0.000 

JOINT  OPERATIONS 

Yes 

0.033 

1.000 

NESI  DEVELOPMENT 

Yes 

0.066 

1.000 

NESI  EVALUATION 

No 

0.066 

0.000 
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4.1.1  Capability  Measures  Scoring 

OPERATIONAL  NEEDS  primarily  requires  the  use  of  the  AV-1  and  SV-5,  although 
information  may  also  be  found  in  the  OV-1,  OV-3,  OV-5,  and  SV-7.  The  AV-1  provides  a  list  of 
OPERATIONAL  NEEDS  upon  which  to  base  the  analysis.  At  a  minimum,  the  AV-1  provides  the 
system  purpose  and  goals.  The  SV-5  is  then  used  to  trace  those  requirements  down  to  functions 
or  components,  thereby  ensuring  that  each  Operational  Need  is  met  by  some  portion  of  the 
system.  The  AV-1  provided  by  the  sponsoring  organization  had  no  information  regarding  the 
operational  needs.  In  addition,  there  was  no  SV-5  from  which  to  trace  functions  to  components. 
Therefore,  the  OPERATIONAL  NEEDS  measure  scores  0%  in  this  evaluation  with  zero  associated 
value. 

THREAT  DETECTION  was  scored  based  on  the  existence  of  a  Threat  Detection  plan.  At  a 
later  point  in  the  development  of  the  project,  the  Threat  Detection  plan  itself  may  be  graded,  but 
at  this  point,  its  existence  was  the  only  important  value.  The  OV-5  was  examined  in  this  case  to 
detennine  its  existence.  OV-1  and  OV-3  were  also  used  for  back  up  and  additional  information 
for  the  Threat  Detection  plan  scoring.  The  OV-5  has  an  operational  activity  devoted  to  the 
DAWDR  concept  of  Detect.  Under  this  activity  are  many  sub  activities  outlining  exactly  how 
the  JFPASS  will  accomplish  the  Detect  activity.  In  addition  to  the  information  in  the  OV-5,  the 
OV-1  shows  a  system  component  devoted  to  threat  detection  as  well.  The  THREAT  DETECTION 
measure  scored  “Yes”  with  an  associated  value  of  one. 

The  same  architectural  views  were  used  in  the  evaluation  of  THREAT  ASSESSMENT  as 
were  used  for  THREAT  DETECTION.  The  OV-5  included  a  similar  activity  called  “Assess,”  which 
accomplished  the  activities  required  in  a  Threat  Assessment  Plan.  The  OV-1  also  includes  an 
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assessment  component  in  the  system  scope.  THREAT  ASSESSMENT  scored  “yes”  in  the 
evaluation  with  an  associated  value  of  one. 

The  OV-5  and  OV-1  were  again  employed  in  the  scoring  of  WARNING  PLAN.  The  OV-5 
includes  a  Warn  activity.  The  OV-1  also  includes  a  “Wide  Area  Alert”  component.  There  are 
some  other  aspects  of  the  OV-1  which  allude  to  a  Warning  plan,  including  the  “Chemical 
Sensors,”  “Lan,”  and  “C2,”  all  of  which  may  accomplish  a  portion  of  the  warning  plan. 

WARNING  PLAN  scored  “Yes”  in  the  evaluation  with  an  associated  value  of  one. 

The  SV-8  view  typically  includes  infonnation  related  to  the  Flexibility  of  a  system. 

Since  an  SV-8  does  not  yet  exist  for  the  JFPASS,  other  views  were  examined  for  infonnation 
regarding  ADAPTION.  No  information  was  found  to  score  ADAPTION  in  the  existing  architecture 
products.  ADAPTION  was  scored  as  “static”  in  this  evaluation.  A  flexible  system  must  include 
information  regarding  flexibility,  as  well  as  how  the  system  may  be  altered  in  the  architecture.  A 
“Static”  score  yields  zero  value  in  this  hierarchy. 

The  Technology  Readiness  Level  (TRL)  Scale  was  employed  for  the  TECHNOLOGICAL 
AVAILABILITY  measure.  Typically,  an  SV-9  would  include  the  necessary  infonnation  to 
detennine  the  TRL  of  a  component.  Since  there  was  no  SV-9  product,  specific  infonnation  on 
system  components  was  searched  for  among  the  existing  products.  The  TRL  was  not  explicitly 
or  implicitly  stated  in  any  of  the  existing  products  nor  was  there  infonnation  to  extrapolate  the 
TRL  for  any  component.  Therefore,  TECHNOLOGICAL  AVAILABILITY  was  scored  as  “TRL  1  or 
No  data”  with  an  associated  value  of  0.05.  The  lowest  level  of  TRL  was  assigned  since  this  level 
accounts  for  components  that  have  not  yet  entered  any  phase  of  development. 

The  ENVIRONMENTAL  IMPACT  measure  was  scored  using  the  TV-1  product.  Section  3.93 
of  the  TV-1  is  the  Environmental  section.  The  guidance  documents  listed  here  include  Mil 
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Standards  for  Environmental  Engineering,  Electromagnetic  Compatibility,  and  Interference 
Standard  requirements.  The  documents  included  represent  a  cross  section  of  some  of  the  types 
of  environmental  guidance  documents  that  must  be  considered  in  the  design  of  an 
environmentally  practical  system.  The  inclusion  of  these  documents  in  the  TV-1  assumes  the 
inclusion  of  these  documents  in  the  design  of  the  system.  The  standards  being  used  in  this 
system  are  military  and  United  States  federal  standards;  therefore,  the  system  was  scored 
“CONUS  and  Contingency  constraints”  with  an  associated  value  of  0.8.  The  inclusion  of 
military  standards  shows  that  contingency  environments  have  been  considered. 

Indicators  for  the  MONETARY  PRACTICALITY  -  INITIAL  measure  may  be  found  in  the  OV- 
5  product,  although  there  are  no  specifically  required  views  in  DoDAF  for  estimates.  Cost  may 
be  included  in  a  layer  of  the  OV-5  for  some  tools.  If  no  cost  is  included  in  the  OV-5,  the  OV-7  is 
also  examined  for  possible  cost  elements.  The  JFPASS  architecture  had  no  cost  infonnation  in 
any  of  the  available  views;  therefore,  the  MONETARY  PRACTICALITY  measure  scored  “Cost 
Unknown”  with  an  associated  value  of  zero. 

MONETARY  PRACTICALITY  -  MAINTENANCE  measure  also  scored  “Cost  Unknown”  in 
this  evaluation.  The  same  views  were  examined  for  costs  elements,  but  JFPASS  did  not  include 
any  cost  infonnation.  The  life-cycle  cost  for  the  system  is  an  important  consideration  for 
decision-makers,  but  this  estimate  was  not  available  in  the  provided  views,  so  the  value 
associated  with  MONETARY  PRACTICALITY  -  MAINTENANCE  was  zero. 

4,1.2  Maintainability  Measures  Scoring  Score 

Every  system  must  be  designed  with  tolerances  for  components  to  ensure  that  the  system 
is  within  design  parameters.  DoDAF  includes  a  vehicle  for  these  requirements  by  way  of  the 
SV-7  product.  Since  the  JFPASS  does  not  currently  include  an  SV-7;  therefore,  the  existence  of 
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design  standards  for  supportability  cannot  be  verified.  None  of  the  other  provided  products 
include  the  information  either.  SUPPORTABILITY  REQUIREMENTS  therefore  scores  “No”  with  an 
associated  value  of  zero. 

Reliability  Requirement  considerations  must  also  be  taken  into  account  in  the  design  of  a 
system.  Since  the  SV-7  does  not  yet  exist  in  the  JFPASS,  the  equations  and  values  required  to 
ensure  the  inclusion  of  Reliability  could  not  be  located.  The  architecture  must  include  some 
verification  of  these  items  to  ensure  that  they  are  considered  in  the  system  design.  RELIABILITY 
REQUIREMENTS  scored  “No”  with  an  associated  value  of  zero  in  this  evaluation. 

It  is  assumed  that  if  redundancy  exists  of  critical  systems,  these  systems  will  be  more 
survivable  during  an  attack.  SYSTEM  REDUNDANCY  ensures  more  system  up-time  during  hostile 
action.  The  OV-6  event  trace  shows  some  evidence  of  system  redundancies.  The  JFPASS 
includes  one  OV-6c  for  a  single  system  activity.  This  view  shows  some  evidence  of  redundancy 
in  this  activity.  Additional  views  show  some  evidence  of  redundancy  as  well.  Based  on  the  data 
provided  in  these  views,  SYSTEM  REDUNDANCY  was  scored  as  “Some  Systems,  Single 
Redundancy.”  This  gave  the  SYSTEM  REDUNDANCY  measure  a  value  of  0.25. 

The  lack  of  an  SV-7  product  does  not  enable  the  verification  of  RECOVERABILITY 
REQUIREMENTS.  These  equations  must  be  verified  to  ensure  their  inclusion  in  the  architectural 
design.  Since  they  cannot  be  verified,  this  measure  scored  “No”  with  an  associated  value  of 
zero. 

4,1.3  Interoperability  Measures  Scoring 

The  JOINT  OPERATIONS  measure  is  based  on  the  consideration  of  all  service  components. 
Since  it  is  a  critical  requirement  that  no  service  be  left  out,  this  measure  is  scored  either  “yes”  or 
“no.”  In  this  instance,  JOINT  OPERATIONS  scored  “yes”  based  on  information  drawn  from  a 
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variety  of  views.  The  AV-1,  OV-2,  OV-3,  OV-4,  and  SV-2  were  the  primary  resources  used  for 
this  evaluation.  The  AV-1  specifically  contains  references  to  all  service  documentation.  In  the 
available  Operational  Views,  there  is  evidence  of  other  service  requirements,  such  as  port 
security  and  convoy  security  which  alludes  to  specific  service  requirements  and  operations.  The 
associated  value  with  this  measure  is  one. 

The  TV-1  is  the  primary  view  employed  in  the  evaluation  of  the  NESI  DEVELOPMENT 
measure.  Section  3.1  and  3.2  of  the  TV-1  refer  to  Infonnation  Technology.  Specifically,  Section 
3.1.2,  Common  Infrastructure  Data  Fonnat  Standards,  and  section  3.1.3,  Network  Management, 
relate  to  the  concept  of  net-centricity.  There  are  also  documents  related  to  wireless 
communications.  While  the  Net-Centric  Standards  for  Interoperability  (NESI)  is  not  referenced 
specifically  in  the  TV-1,  the  concepts  set  forth  in  NESI  are  accounted  for  based  on  the 
documents  provided  in  the  TV-1.  NESI  DEVELOPMENT  scored  “Yes”  with  an  associated  value  of 
one  in  this  evaluation. 

The  evaluation  criterion  and  checklists  provided  in  the  NESI  Guidance  were  not 
completed  for  this  system.  There  is  no  evidence  in  the  provided  architecture  that  the  system  was 
evaluated  for  compliance  with  NESI.  NESI  EVALUATION  scored  “No”  in  this  evaluation  with  an 
associated  value  of  zero. 

4.2  Deterministic  Analysis 

With  all  measures  scored,  a  deterministic  analysis  was  perfonned  to  find  a  single, 
aggregate  score  for  the  entire  project.  Using  the  general  additive  value  function,  this  score 
combines  the  associated  values  for  each  score  as  well  as  the  weight  of  each  measure.  This  score 
is  simply  a  weighted  average  of  these  numbers.  The  Hierarchy  Builder©  software  was  used  to 
generate  graphs  and  do  a  comparative  analysis  of  possible  alternatives. 
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4.2.1  Additive  Value  Function 


The  general  additive  value  function  was  used  to  create  the  JFPASS  final  score.  The 
general  additive  value  function  is  represented  as  (Kirkwood,  1997,  p.  230): 

»W=  I?=1  ^9  4.2 

Where  v(x)  =  the  overall  score  of  the  alternative,  vfxi)  =  the  value  of  the  score  on  the  ith 
measure,  AL  =  weight  of  the  ith  measure,  n  =  the  total  number  of  measure,  £f=i  =  1.0 

Table  4.3  shows  all  measures  and  weights  with  the  Architecture  Quality  measures 
included.  To  obtain  a  score  for  the  full  system,  the  measures  for  Architecture  Quality  must  also 
be  used.  It  is  possible  to  examine  the  two  branches  separately,  but  this  will  give  the  decision¬ 
maker  an  incomplete  picture  of  the  entire  system.  The  detenninistic  analysis  for  the  System 
Effectiveness  branch  of  JFPASS  was  completed  using  the  entire  system  as  well  as  the  System 
Effectiveness  branch  alone.  For  all  analyses,  the  Architecture  Quality  score  was  held  static  from 
Cotton  and  Haase’s  (2009)  evaluation  of  the  system. 
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Table  4.3  All  Measures 


Measure  Description 


OPERATIONAL  NEEDS _ 

THREAT  DETECTION _ 

THREAT  ASSESSMENT _ 

WARNING  PLAN _ 

TECHNOLOGICAL  AVAILABILITY _ 

ENVIRONMENTAL  IMPACT _ 

MONETARY  PRACTICALITY  -  INITIAL _ 

MONETARY  PRACTICALITY  -  MAINTENANCE 

ADAPTATION _ 

SUPPORT  ABILITY  REQUIREMENTS _ 

RELIABILITY  REQUIREMENTS _ 

SYSTEM  REDUNDANCY _ 

RECOVERABILITY  REQUIREMENTS _ 

JOINT  OPERATIONS _ 

NESI  DEVELOPMENT _ 

NESI  EVALUATION _ 

ACCESS _ 

PRODUCT  LOCAT ABILITY _ 

DOCUMENT  PROTECTION _ 

PROTECTION _ 

FILE  MANAGEMENT _ 

FILE  FORMAT _ 

CONNECTIONS _ 

ARCHITECTURE  REDUNDANCY _ 

ARCHITECTURE  ECONOMY _ 

OV  READABILITY _ 

SV  READABILITY _ 

SCALE _ 

DECOMPOSITION _ 

TOOL  FORMAT _ 

DODAF  COMPLIANCY _ 

REQUIREMENT  TRACEABILITY _ 

INTERNAL  CONSISTENCY _ 

EXTERNAL  CONSISTENCY _ 

SME  INVOLVEMENT _ 

SME  EFFECTIVENESS 


and  Weights 


Global  Weight  (A/) 


0.041 


0.020 

0.020 

0.020 

0.020 

0.027 


0.064 

0.040 

0.026 


0.066 

0.066 

0.022 


0.029 

0.029 

0.024 

0.024 

0.012 


Value 


0.000 


0.016 

0.000 

0.000 

0.000 

0.000 

0.000 

0.010 

0.000 


0.066 

0.000 

0.022 


0.000 


0.027 


0.014 

0.024 

0.012 


0.000 

0.009 

0.009 

0.000 

0.000 
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4.2.2  JFPASS  Analysis 

The  final  score  for  the  total  system  was  found  to  be  0.538  out  of  1 .000.  Figure  4. 1  shows 
a  stacked  bar  chart  of  each  of  the  two  major  Tier  1  values.  The  System  Effectiveness  branch, 
though  weighted  higher  (0.6)  in  the  fundamental  objective,  received  less  of  the  overall  weight. 


JFPASS  Current  System 


□  System  Effectiveness  □  Architecture  Quality 


Figure  4.1  JFPASS  Score  Fundamental  Objective  -  Values 


Figure  4.2  shows  the  VDEA  Score  score  sorted  by  measure.  In  this  example,  each 
measure’s  value  and  global  weight  accounts  for  a  portion  of  the  bar  chart.  While  Purposefulness 
is  the  highest  weighted  value,  it  has  four  measures  intended  to  measure  the  concept.  Therefore, 
each  of  those  four  measures  have  a  smaller  global  weight.  NESI  DEVELOPMENT  and  NESI 
EVALUATION,  however,  have  only  one  measure  to  detennine  the  degree  of  attainment  of  the 
value;  therefore,  their  measures’  global  weights  are  higher.  The  high  global  weight  and 
associated  value  of  NESI  DEVELOPMENT  and  NESI  EVALUATION  cause  these  measures  to  be 
among  the  highest  weighted  measures.  It  is  evident  in  Figure  4.2  that  a  large  portion  of  value 
was  lost  due  to  the  lack  of  a  performing  a  NESI  EVALUATION  and  the  lack  of  OPERATIONAL 
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NEEDS  attainment.  Three  of  th  four  Maintainability  measures  rank  third,  sixth,  and  seventh  in 
possible  global  weight  of  the  system.  Some  value  was  earned  by  SYSTEM  REDUNDANCY,  but  no 
value  was  earned  by  Reliability  or  SUPPORT  ABILITY  REQUIREMENTS. 

Of  the  0.60  total  possible  value  of  the  fundamental  objective,  the  System  Effectiveness 
branch  of  the  hierarchy  earned  41.3%  of  its  value.  This  equates  to  0.248  of  the  0.600  possible 
value  units  for  System  Effectiveness.  Figure  4.3  shows  the  measures  under  the  System 
Effectiveness  branch  only.  This  figure  and  score  does  not  take  into  account  any  of  the 
Architecture  Quality  measures  or  scores. 


JFPASS  Current  System 


ONESI  Development 

□  Reliability  Requirements 
□Threat  Detection 
□WarningPlan 
□Supporta  bi  I  ity  Requirements 

□  Document  Protection 

□  DoDAF  Compliancy 
□SV  Readability 

□  Recoverability  Requirements 

□  Decomposition 

□  File  Management 
□Technological  Availability 

□  Monetary  Practicality-  Initial 

□  Requirement  Traceability 
□SME  Involvement 
□Architecture  Redundancy 
□Tool  Format 

□  Internal  Consistency 


■  NESI  Evaluation 

■  Operational  Needs 

□  Threat  Assessment 

■  Redundancy 

□  Access  Control 

□  JointOperations 
fflOVReadability 

□  Adaptation 

□  Scale 

□  Access 

□  FileFormat 

□  Environmental  Impact 

□  Monetary  Practicality  -  Maintenance 

□  SME  Effectiveness 

□  Connections 

□  Architecture  Economy 

□  Product  Locatability 

□  External  Consistency 


Figure  4.2  JFPASS  Score  Fundamental  Objective  -  Measures 
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JFPASS  Current  System 


□  NESI  Development 

□  Reliability  Requirements 

□  Threat  Detection 

□  Warning  Plan 

□  Supportability  Requirements 

□  Adaptation 

□  Technological  Availability 

□  Monetary  Practicality  -  Initial 


■  NESI  Evaluation 

■  Operational  Needs 

□  Threat  Assessment 

□  Redundancy 

□  Joint  Operations 

□  Recoverability  Requirements 

□  Environmental  Impact 

□  Monetary  Practicality  -  Maintenance 


Figure  4.3  System  Effectiveness  Score  -  Measures 


When  examining  the  System  Effectiveness  branch  alone  in  figure  4.3,  it  is  evident  where 
value  was  lost  in  the  system.  NESI  EVALUATION,  RELIABILITY  REQUIREMENTS,  and 
OPERATIONAL  NEEDS  are  the  three  highest  ranking  measures  that  did  not  score  any  value. 
SYSTEM  REDUNDANCY  earned  0.25  of  its  total  possible  value,  but  none  of  the  other 
Maintainability  measures,  SUPPORTABILITY  and  RECOVERABILITY  REQUIREMENTS,  earned  any 
value.  JOINT  OPERATIONS  earned  full  value,  but  ADAPTATION,  TECHNOLOGICAL  AVAILABILITY, 
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MONETARY  PRACTICALITY  -  INITIAL,  and  MONETARY  PRACTICALITY  -  MAINTENANCE  did  not 


earn  any  value. 

The  next  area  of  the  hierarchy  to  be  analyzed  was  the  Capability  branch  of  the  System 
Effectiveness  branch.  Capability  earned  5 1 .4%  of  its  total  possible  value  (45%  of  System 
Effectiveness ).  The  measures  providing  value  to  this  branch  are  THREAT  DETECTION,  THREAT 
ASSESSMENT,  WARNING  PLAN,  TECHNOLOGICAL  AVAILABILITY,  and  Environmental 
Practicality.  Figure  4.4  shows  the  bar  chart  of  this  branch.  The  Interoperability  branch  and 
Maintainability  branch  each  only  contributed  one  measure  worth  of  value  to  the  total  score. 
Additional  bar  charts  for  the  System  Effectiveness  Branch  are  found  in  Appendix  C. 


JFPASS  17  February  2009  0.514 


□  Operational  Needs 

□  Threat  Assessment 

□  Adaptation 

□  Environmental  Impact 

□  Monetary  Practicality  -  Maintenance 


■  Threat  Detection 

□  Warning  Plan 

□  Technological  Availability 

□  Monetary  Practicality  -  Initial 


Figure  4.4  Capability  Score  -  Measures 
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4.2.3  Gap  Analysis 


The  actual  generation  of  alternatives  was  completed  via  a  gap  analysis.  This  gap  analysis 
allows  the  model  builder  to  detennine  how  much  of  an  affect  each  measure  has  on  the  overall 
score.  By  adjusting  the  score  of  each  measure  and  evaluating  the  total  system  score,  the  gap 
between  the  current  value  and  highest  possible  (or  lowest  possible)  value  can  be  found.  Table 
4.4  shows  the  results  of  this  sensitivity  analysis  in  the  range  of  each  measure’s  scoring 
possibilities.  The  range  is  shown  in  the  low  and  high  columns.  The  “current”  column  shows  the 
current  overall  system  score,  with  the  final  column  showing  the  gap  between  the  current  system 
score  and  the  highest  possible  score  that  could  be  attained  by  maximizing  the  measure  in 
question. 


Table  4.4  Gap  Analysis  Score  Ranges 


Global  Value 

Measure 

Low 

High 

Current 

Gap 

OPERATIONAL  NEEDS 

0.538 

0.578 

0.538 

0.040 

THREAT  DETECTION 

0.497 

0.538 

0.538 

0.041 

THREAT  ASSESSMENT 

0.497 

0.538 

0.538 

0.041 

WARNING 

0.497 

0.538 

0.538 

0.041 

TECHNOLOGICAL 

AVAILABILITY 

0.538 

0.557 

0.538 

0.019 

ENVIRONMENTAL  IMPACT 

0.521 

0.542 

0.538 

0.021 

MONETARY  PRACTICALITY  - 
INITIAL 

0.538 

0.558 

0.538 

0.020 

MONETARY  PRACTICALITY  - 
MAINTENANCE 

0.538 

0.558 

0.538 

0.020 

ADAPTATION 

0.538 

0.565 

0.538 

0.027 

JOINT  OPERATIONS 

0.505 

0.538 

0.538 

0.033 

NESI  DEVELOPMENT 

0.472 

0.538 

0.538 

0.066 

NESI  EVALUATION 

0.538 

0.604 

0.538 

0.066 

SUPPORT  ABILITY 

REQUIREMENTS 

0.538 

0.572 

0.538 

0.034 

RELIABILITY  REQUIREMENTS 

0.538 

0.602 

0.538 

0.064 

SYSTEM  REDUNDANCY 

0.528 

0.567 

0.538 

0.039 

RECOVERABILITY 

REQUIREMENTS 

0.538 

0.564 

0.538 

0.026 
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4.2.3. 1  Capability  Measurement  Sensitivity 

The  OPERATIONAL  NEEDS  measurment  was  altered  by  increasing  the  score  by  10%  or  0.1 
incrementally.  Each  incremental  increase  created  a  new  alternative.  The  ten  created  alternatives 
showed  a  gap  of  0.04  value  units  between  the  lowest  score  of  OPERATIONAL  NEEDS  and  the 
highest  score. 

The  THREAT  DETECTION  measure  is  a  binary  measure;  therefore,  there  are  only  two 
scoring  possibilities.  Since  the  current  score  is  set  to  “yes,”  the  alternative  to  this  measurement 
scenario  is  if  the  measurement  was  scored  “no.”  The  alternative  was  worth  less  value  than  the 
current  situation,  with  a  loss  of  0.041  value  units.  THREAT  ASSESSMENT  is  the  same  in  all  ways 
to  the  THREAT  DETECTION  measurement  in  sensitivity.  The  range  is  also  0.041,  as  well  as  the 
binary  nature.  The  WARNING  PLAN  measure  is  also  similar  to  THREAT  DETECTION  and  THREAT 
ASSESSMENT.  It  also  has  a  range  of  0.041  meaning  that  if  it  were  to  be  scored  “no,”  the 
maximum  global  system  value  lost  would  be  0.041. 

TECHNOLOGICAL  AVAILABILITY  had  a  total  of  nine  generated  alternatives  for  sensitivity 
analysis.  The  TECHNOLOGICAL  AVAILABILITY  measurement  was  set  to  each  TRL  level  for  each 
alternative  and  the  range  for  the  TECHNOLOGICAL  AVAILABILITY  measure  was  found.  The  range 
was  0.019  change  in  value  unit  from  the  lowest  score  to  the  highest  score  of  TECHNOLOGICAL 
AVAILABILITY.  The  current  score  of  this  measure  was  “Unknonwn/TRL  1,”  therefore  any 
additional  infonnation  or  change  to  this  measure  would  cause  an  increase  to  the  total  system 
score. 

ENVIRONMENTAL  IMPACT  measurement  has  four  possible  score  settings.  This  gives 
ENVIRONMENTAL  IMPACT  a  range  of  0.021,  from  0.521  to  0.538  total  system  scores.  The  current 
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system  score  is  0.538  with  only  one  higher  score  possible.  Therefore,  only  0.004  value  units  are 
available  for  ENVIRONMENTAL  IMPACT. 

MONETARY  PRACTICALITY  -  INITIAL  has  four  possible  score  categories.  The  current 
score  of  this  measure  earns  no  value,  therefore  any  change  to  the  score  would  cause  an  increase 
to  the  total  system  score.  The  possible  change  is  0.20  globally.  Figure  4.32  and  4.33  show  the 
global  and  local  measure  sensitivity.  MONETARY  PRACTICALITY  -  MAINTENANCE  has  the  same 
range  and  numerical  value  possibilities  as  MONETARY  PRACTICALITY  -  INITIAL. 

ADAPTION  has  five  possible  value  categories.  Currently  the  ADAPTION  measure  earns  no 
value,  therefore  any  increase  in  measurement  will  cause  an  increase  in  total  system  score.  The 
range  of  scores  for  ADAPTION  is  0.027  globally.  While  ADAPTION  has  a  smaller  effect  on  the 
total  system  score,  any  value  earned  helps  the  overall  system  score. 

4.2.3.2  Maintainability  Measurement  Sensitivity 

SUPPORT  ABILITY  REQUIREMENTS  is  a  binary  measure,  which  is  currently  measured  at  its 
lower  value  possibility.  If  the  score  is  set  to  “yes,”  the  possible  change  in  value  is  0.034  value 
units.  This  change  in  overall  score  indicates  that  this  measurement  will  have  a  significant  impact 
on  the  overall  score. 

The  RELIABILITY  REQUIREMENTS  measure  is  also  a  binary  scoring  set.  Since  the 
Reliability  value  has  a  higher  global  weight,  the  range  in  score  change  is  larger  for  this  measure 
of  Maintainability.  The  VDEA  score  range  is  0.064  for  RELIABILITY  REQUIREMENTS. 

RECOVERABILITY  REQUIREMENTS  is  also  a  binary  measure.  It  is  similar  to  both 
SUPPORT  ABILITY  REQUIREMENTS  and  RELIABILITY  REQUIREMENTS  and  is  currently  scored  as 
“no.”  The  range  for  this  measure  is  0.538  -  0.564,  meaning  that  the  highest  possible  value 
change  between  the  lower  score  and  higher  score  is  0.026. 
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The  SYSTEM  REDUNDANCY  measure  is  rated  on  a  scale  of  four  possible  levels.  The 
current  system  is  scored  at  the  second  level.  The  range  between  the  lowest  and  highest  scoring 
possibilities  is  0.039  value  units.  As  one  of  the  four  Maintainability  measures,  SYSTEM 
REDUNDANCY  is  an  important  design  consideration,  which  is  evident  in  its  effect  on  the  total 
system  score. 

4.2.3.3  Interoperability  Measures  Sensitivity 

The  JOINT  OPERATIONS  measurement  scored  “yes”  in  this  evaluation,  giving  it  its  full 
possible  value.  If  it  had  been  scored  no,  the  greatest  change  to  the  total  system  score  would  have 
been  0.033  value  units,  taking  the  system  score  to  0.505.  This  is  a  relatively  large  change  for  a 
single  measure,  indicating  that  the  JOINT  OPERATIONS  measurement  is  of  great  importance. 

The  NESI  DEVELOPMENT  measurement  is  a  binary  measure,  currently  scored  at  its  higher 
score  of  “yes.”  NESI  DEVELOPMENT  is  one  of  the  more  important  measures  in  this  evaluation, 
with  a  global  weight  of  0.066.  Therefore,  its  impact  on  the  total  score  is  higher  than  most  other 
measures.  The  swing  between  its  higher  and  lower  score  possibilities  is  0.066  value  units.  This 
makes  the  lower  possible  range  of  the  VDEA  score  to  0.472  for  this  measure. 

The  final  measure  on  the  System  Effectiveness  branch  of  the  hierarchy  was  NESI 
EVALUATION.  This  measure  also  has  a  high  global  weight  making  it  a  relatively  important 
measure.  In  this  case,  the  system  scored  at  the  lower  possibility,  therefore  if  the  score  was 
changed  to  “yes,”  the  change  in  VDEA  score  would  be  0.066,  taking  the  total  score  to  0.604. 
4.2.4  Alternative  Generation 

Step  six  of  the  Value-Focused  Thinking  (VFT)  process  is  the  Alternative  Generation  step. 
In  the  standard  VFT  process,  at  this  point,  the  hierarchy  would  be  used  to  find  alternatives  which 
fit  the  evaluation  criteria.  In  the  case  of  the  Joint  Force  Protection  Advanced  Security  System 
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(JFPASS),  though,  a  single  alternative  was  created  by  the  decision-maker.  However,  several 
additional  alternatives  were  generated  as  comparison  criteria  for  the  decision-maker.  These 
alternatives  also  demonstrated  the  additional  insight  to  be  gained  from  the  VFT  process. 

When  generating  alternatives,  two  approaches  were  taken.  First,  a  set  of  alternatives  was 
generated  which  represented  the  baseline  or  current  system  with  improvements.  These 
alternatives  show  future  iterations  of  the  system  with  a  product  focus,  in  which  new  products  are 
added  to  the  architecture  with  the  intent  of  improving  the  VDEA  score.  The  second  set  of 
alternatives  represents  random  scoring  scenarios.  These  scenarios  do  not  take  into  account  the 
current  or  baseline  architecture. 

Table  4.5  shows  the  alternatives  that  were  generated  for  the  decision-maker.  These 
alternatives  represent  different  versions  of  the  architecture.  In  Table  3.8,  each  measure  is  linked 
to  its  source  views.  The  “Perfect  Architecture”  represents  an  architecture  that  scores  100%  value 
in  all  currently  available  System  Effectiveness  areas  as  an  upper  bound.  The  “Current 
Architecture”  or  “Baseline”  is  the  architecture  as  provided  by  the  sponsor.  Other  alternatives 
represent  possible  alterations  to  the  existing  architecture,  based  on  the  addition  of  more  views. 
The  views  used  for  additional  alternatives  were  determined  based  on  the  sources  for  measures 
(Table  4.6)  and  the  gap  analysis  shown  in  Figure  4.5.  By  determining  where  the  current 
architecture  lost  value,  it  was  possible  to  generate  alternatives  that  filled  those  gaps  by  adding  the 
necessary  views.  For  example,  the  OPERATIONAL  NEEDS  measure  requires  at  a  minimum  an 
AV-1  and  SV-5,  in  addition  to  all  existing  views,  to  earn  full  value.  Therefore,  an  alternative 
was  created  for  a  full  AV-1  and  SV-5,  which  assumes  that  these  products  include  the  minimum 
information  necessary  to  assign  full  value  to  OPERATIONAL  NEEDS. 
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Table  4.5.  Generated  Alternatives 


Perfect  Architecture 

Baseline 

Baseline  plus  OV-3 

Baseline  plus  SV-5 

Baseline  plus  SV-7 

Baseline  plus  SV-8 

Baseline  plus  SV-9 

Baseline  plus  full  AV-1 

Baseline  plus  AV-1  and  SV-7 

Baseline  plus  AV-1,  SV-7,  and  SV-5 

Baseline  plus  SV-7,  SV-8,  and  SV-9 

Random  VDEA  Score  1 

Random  VDEA  Score  2 

Random  VDEA  Score  3 

Random  VDEA  Score  4 

Table  4.6.  Required  Views  for  Measures  (Osgood,  2009) 


AV-1  Overview  and  Summary 
Information 

OV-1  High-Level  Operational 
Concept  Graphic 

OV-2  Operational  Node 
Connectivity  Description 

OV-3  Operational  Information 
Exchange  Matrix 

OV^f  Organizational  Relationship 

Chart 

1 

OV-6  Operational  Rules 
Model/State  Transition/Event- 
Trace  Description 

SV-2  System/Services 
Communication  Description 

SV-5  Operational  Activity  to  System 
Function/Systems/Services 
Traceability  Matrix 

SV-7  Systems/Services 
Performance  Parameters  Matrix 

SV-8  Systems/Services  Evolution 

Description 

SV-9  Systems/Services  Technology 

Forecast 

1 

Operational  Needs 

X 

X 

X 

X 

X 

X 

Threat  Detection 

X 

X 

■ 

Threat  Assessment 

X 

X 

Warning  Plan 

X 

X 

■ 

Technological  Availability 

X 

Environmental  Impact 

■ 

Monetary  Practicality  -  Initial 

X 

Monetary  Practicality  -  Maintenance 

m 

Adaptation 

X 

Suppo liability  Requirements 

X 

Reliability  Requirements 

X 

System  Redundancy 

X 

Recoverability  Requirements 

X 

Joint  Operations 

X 

X 

X 

X 

X 

NESI  Development 

X 

NESI  Evaluation 

X 
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Figure  4.5.  Gap  Analysis 
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Monetary  Practicality  -  Maintenance  ■  NESI  Development  NESI  Evaluation  I  Operational  Needs 


Baseline  plus  SV-7  (in  addition  to  existing  products)  is  an  alternative  created  to  show  the 
difference  in  score  if  the  Maintenance  value  measures  (except  SYSTEM  REDUNDANCY)  were 
assigned  full  values.  Without  an  SV-7,  it  is  very  difficult  to  score  these  measures.  The  OV-3 
alternative  exists  since  it  is  a  supporting  view  for  four  measures.  Baseline  plus  SV-5 
demonstrates  the  effect  of  adding  only  an  SV-5  without  a  full  AV-1.  Baseline  plus  SV-8  shows 
the  value  associated  with  adding  an  SV-8  and  Baseline  plus  SV-9  shows  the  value  associated 
with  adding  an  SV-9.  The  SV-8  and  SV-9  products  account  for  the  TECHNOLOGICAL 
AVAILABILITY  and  ADAPTION  measures.  An  alternative  was  also  created  that  incorporates  a  full 
AV-1,  SV-7,  and  SV-5,  which  allows  the  hierarchy  to  maximize  the  OPERATIONAL  NEEDS 
measure,  as  well  as  three  of  the  Maintainability  measures.  Finally,  an  alternative  was  generated 
that  includes  an  SV-7,  SV-8,  and  SV-9.  This  alternative  represents  a  situation  in  which 
TECHNOLOGICAL  AVAILABILITY,  ADAPTATION,  and  three  of  the  four  Maintainability  measures. 
The  full  scoring  scheme  for  each  of  these  alternatives  is  shown  in  Appendix  D. 

The  second  set  of  generated  alternatives  was  produced  by  allowing  each  measure’s  score 
to  take  on  a  random  value.  In  cases  where  the  measurement  was  on  a  continuous  interval,  a 
random  number  was  selected  on  that  interval.  In  cases  where  the  measure  could  take  on  discrete 
categorical  values,  each  category  was  given  equal  likelihood  of  occurrence  for  selection.  Once 
all  scoring  scenarios  were  generated,  the  related  value  was  summed  to  obtain  an  cumulative 
score.  This  process  was  perfonned  500  times  and  four  of  these  500  cases  are  presented  as 
alternatives.  Since  measurements  are  independent  and  mutually  exclusive,  each  measurement 
score  may  take  on  any  scoring  value  without  an  effect  on  the  measurement  of  other  values. 

These  alternatives  produced  from  Monte  Carlo  sampling  are  intended  to  provide  additional 
comparison  criteria  for  the  baseline. 
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Monte  Carlo  generation  of  alternatives  was  completed  using  a  spreadsheet.  To  obtain  the 
composite  random  scores,  the  Architecture  Quality  scores  were  held  static  with  their  current 
values.  A  Monte  Carlo  sampling  technique  was  used  to  randomize  System  Effectiveness 
measurements.  This  measurement  was  then  transformed  to  a  value  using  the  SDVF  and 
multiplied  by  its  measure  weight  (T£).  The  composite  score  was  for  each  alternative  was 
calculated  by  summing  all  values  for  each  measure.  Five  hundred  separate  scoring  iterations 
were  created.  These  500  new  “alternatives”  represented  500  different  possible  alternatives  with 
random  measurements  for  each  measure,  and  therefore  a  random  score.  They  do  not  represent 
the  current  JFPASS  value  or  measurement,  only  hypothetical  VDEA  scores.  Once  the  set  of  500 
scores  were  produced,  a  random  number  generator  was  used  to  select  four  of  the  iterations  from 
the  set  of  500.  Each  of  the  numbers  selected  represented  a  random  score  of  the  VDEA 
instrument.  The  random  selector  chose  random  alternatives  26,  207,  379,  and  4201;  hereafter 
called  Random  VDEA  Score  alternatives.  Each  of  these  situations  were  then  entered  to  the 
Hierarchy  Builder©  software  and  analyzed  alongside  the  other  generated  alternatives. 

4.2.5  Alternative  Analysis 

As  a  part  of  the  JFPASS  detenninistic  analysis,  each  of  the  generated  alternatives  were 
also  examined.  Appendix  D  shows  the  scores  for  each  of  the  measures  in  each  alternative. 

Figure  4.6  shows  the  rankings  and  scores  of  all  alternatives.  It  is  important  to  note  that  though 
the  addition  of  certain  views  may  not  help  the  overall  score  of  the  system,  that  does  not  mean 
that  those  views  at  not  useful.  Each  view  within  DoDAF  serves  a  specific  purpose  and  should 
not  be  discounted  if  the  information  contained  within  it  is  helpful  to  the  system  designers  or 
decision-maker.  This  analysis  is  based  on  the  information  that  is  important  the  scoring  of  the 
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system  based  on  the  decision-maker’s  values.  If  additional  views  are  required,  they  may  not 
impact  the  score  of  the  system,  but  this  does  not  lessen  their  importance  to  system  design. 


JFPASS  Perfect  System  0.890 
Baselineplus  SV-7, SV-8,  and  SV-9  0.709 
Basel i  ne  pi  us  AV-l,SV-7,  andSV-5  0.704 
RandomVDEA  Score 4  0.686 
Baselineplus SV-7  0.663 
RandomVDEA  Score 3  0.583 
Baselineplus  AV-landSV-5  0.578 
Baselineplus SV-8  0.565 
Baselineplus  SV-9  0.557 
RandomVDEA  Score  2  0.540 
Baselineplus OV-5  0.538 
Baselineplus OV-3  0.538 
Baselineplus  Full  AV-1  0.538 
Baseline  0.538 
RandomVDEA  Score  1  0.454 
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□  NESI  Development 

□  Rel  i  a  bi  I  ity  Requi  rements 
□Threat  Detection 
□WarningPlan 
□Supportabi I  ity  Requirements 

□  Document  Protection 

□  DoDAF  Compliancy 
□SV  Readability 

□  Recovera  bi  I  i  ty  Requi  rements 

□  Decomposition 

□  FileManagement 
□Technological  Availability 

□  Monetary  Practicality-Initial 

□  Requirement  Traceability 
□SME  Involvement 
□Architecture  Redundancy 
□Tool  Format 

□  Internal  Consistency 


■  NESI  Evaluation 

□  Operational  Needs 

□  Threat  Assessment 

■  Redundancy 

□  Access  Control 

□  JointOperations 
BOV  Readability 

□  Adaptation 

□  Scale 

□  Access 

□  FileFormat 

□  Environmental  Impact 

□  Monetary  Practicality-  Maintenance 

□  SME  Effectiveness 

□  Connections 

□  Architecture  Economy 

□  Product  Locatability 

□  External  Consistency 


Figure  4.6  All  Alternatives  and  Scores 


The  Perfect  System  score  represents  a  system  in  which  all  System  Effectiveness  measures 
have  scored  their  maximum  value.  The  Architecture  Quality  scores  were  held  constant  from 
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Cotton  and  Haase’s  (2009)  evaluation  of  the  system.  With  all  System  Effectiveness  scores 
maximized,  the  system  scored  0.890  of  1.000  possible  value  units. 

The  highest  scoring  “non-perfect”  alternative  was  a  situation  in  which  SV-7,  SV-8,  and 
SV-9  products  are  added.  The  addition  of  these  views  allows  for  the  maximization  of 
TECHNOLOGICAL  AVAILABILITY,  ADAPTATION,  and  three  of  the  four  Maintainability  measures. 
Each  of  these  views  maximizes  different  areas  of  the  hierarchy,  to  show  the  additive  advantages 
of  creating  additional  views.  This  alternative  scored  0.709  of  1.000  possible  value  units. 

The  next  highest  scoring  alternative  was  the  current  system  with  and  AV-1,  SV-7,  and 
SV-5  added.  Adding  these  views  allows  the  system  to  maximize  both  the  OPERATIONAL  NEEDS 
measure  as  well  as  the  Maintainability  measures.  The  addition  of  a  full  AV-1  and  SV-5  allows 
for  OPERATIONAL  NEEDS  to  be  maximized,  while  the  SV-7  allows  for  the  Maintainability 
measures  to  be  maximized.  This  alternative  scored  0.704  out  of  1.000  value  units. 

The  Random  VDEA  Score  4  alternative  represents  a  random  scoring  situation.  The 
individual  scores  for  each  measure  of  Random  alternative  4  are  shown  in  Appendix  D.  Random 
alternatives  were  generated  to  demonstrate  to  the  decision-maker  additional  possible  scoring 
situations  which  may  not  have  been  considered  in  the  generation  of  other  alternatives.  This 
allows  the  decision-maker  other  means  to  achieve  a  certain  level  of  architecture  value. 

JFPASS  with  SV-7  was  the  next  highest  scoring  alternative.  This  alternative  is  meant  to 
demonstrate  the  effect  that  the  addition  of  an  SV-7  that  includes  the  necessary  documentation  for 
the  Maintainability  measures  would  have  on  the  system  score.  SV-7  is  therefore  the  next  most 
important  view  to  be  added  to  the  existing  architecture.  This  view  would  provide  the  required 
information  to  prove  that  the  values  under  Maintainability  were  considered  in  the  system  design. 
The  score  for  this  measure  was  0.663. 
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Random  VDEA  Score  3  scored  the  next  highest  value.  This  alternative  scored  0.583. 

The  JFPASS  alternative  that  includes  AV-1  and  SV-5  allows  the  OPERATIONAL  NEEDS  measure 
to  be  maximized.  By  adding  these  views  with  the  required  information,  the  score  of  the  system 
becomes  0.578.  The  SV-8  product  allows  the  system  to  maximize  its  score  for  ADAPTION.  If 
this  view  is  created,  with  necessary  infonnation  for  an  improved  ADAPTION  score,  the  total 
system  score  increases  from  0.538  to  0.565. 

SV-9  allows  for  a  similar  increase  in  value.  The  addition  of  an  AV-9  view  will  allow  for 
the  TECHNOLOGICAL  AVAILABILITY  measure  to  be  scored  higher.  This  measure  is  more  difficult 
though.  There  is  a  possibility  that  simply  adding  the  SV-9  product  will  not  increase  the  score. 
Since  the  TRL  is  based  on  the  actual  availability  of  technology,  in  order  to  maximize  the  TRL 
scoring,  the  technology  used  in  the  system  must  rate  higher  on  the  TRL  scale.  The  creation  of 
this  alternative  allows  the  decision-maker  to  see  how  much  of  an  effect  the  maximization  of 
TECHNOLOGICAL  AVAILABILITY  will  have  on  the  total  system  score. 

Random  VDEA  Score  2  scored  the  next  highest  on  the  list  of  alternatives.  This  random 
alternative  had  a  score  of  0.540  and  represents  a  system  with  very  minor  changes  to  the  current 
configuration.  In  addition,  this  alternative  shows  the  effects  of  failing  to  maximize  or  earn  value 
on  some  of  the  views  that  the  baseline  has  achieved  value  on. 

The  next  three  alternatives  all  scored  the  same  as  the  current  JFPASS  system.  The 
addition  of  an  SV-5  product  by  itself  does  not  add  any  value  to  the  system.  The  SV-5  is  used  to 
trace  capabilities  to  system  functions,  but  without  a  list  of  OPERATIONAL  NEEDS,  the  SV-5  does 
not  add  much  value  based  on  what  is  important  to  the  decision-maker.  The  SV-5  is  only  useful 
when  combined  with  a  complete  AV-1.  Without  the  SV-5,  though,  a  more  complete  and  explicit 
AV-  1  is  of  little  value  as  well.  The  system  with  an  OV-3  added  also  does  not  earn  any 
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additional  value.  This  was  the  last  view  that  could  possibly  add  value  to  the  system  or  was 
useful  in  the  determination  of  the  scoring  of  some  values.  The  OV-3  is  used  for  the  score  of 
OPERATIONAL  NEEDS,  THREAT  DETECTION,  THREAT  ASSESSMENT,  WARNING  PLAN,  and  JOINT 
OPERATIONS.  Since  the  OV-3  is  only  a  supplemental  view  for  determining  the  score  of  these 
measures,  its  additional  does  not  help  the  current  system.  In  the  case  of  THREAT  DETECTION, 
THREAT  ASSESSMENT,  WARNING  PLAN,  and  JOINT  OPERATIONS,  these  measures  were  already 
scored  at  maximum  value,  therefore  the  OV-3  does  not  assist  in  scoring  those  measures. 

Finally,  the  Random  VDEA  Score  1  Alternative  has  a  VDEA-Score  of  0.454.  This 
random  alternative  represents  a  system  which  is  more  lacking  than  the  existing  system.  This 
random  alternative  lacks  many  of  the  stronger  valued  measures  and  therefore  scores  lower.  This 
alternative  shows  a  contrast  to  the  baseline  in  which  many  of  the  values  did  not  earn  their  full 
values. 

The  scores  and  difference  from  the  current  system  for  all  alternatives  can  be  found  in 
Table  4.7.  This  table  illustrates  exactly  how  much  effect  improvements  to  the  current 
architecture  will  have  on  the  total  score.  For  example,  the  addition  of  an  SV-7  can  cause  a 
maximum  change  of  0.125  value  units  to  the  system.  Therefore,  adding  this  one  product  has  the 
highest  impact  on  the  system.  Adding  an  SV-7,  SV-5,  and  completing  the  AV-1  will  have  a 
maximum  change  of  0. 166  value  units.  But  making  this  change  will  require  the  addition  of  two 
products  and  alterations  to  an  existing  product.  Therefore,  the  most  cost  effective  value  adding 
measure  will  be  to  add  an  SV-7. 
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Table  4.7  Alternative  Scores  and  Maximum  Value  Additions 


Alternative  Name 

Score 

Maximum 
Value  Change 

JFPASS  Perfect  System 

.890 

0.352 

Baseline  plus  SV-7,  SV-8,  and  SV-9 

.709 

0.171 

Baseline  plus  AV-1,  SV-7,  and  SV-5 

.704 

0.166 

Random  VDEA  Score  4 

.686 

0.148 

Baseline  plus  SV-7 

.663 

0.125 

Random  VDEA  Score  3 

.583 

0.045 

Baseline  plus  AV-1  and  SV-5 

.578 

0.040 

Baseline  plus  SV-8 

.565 

0.027 

Baseline  plus  SV-9 

.557 

0.019 

Random  VDEA  Score  2 

.540 

0.002 

Baseline 

.538 

0.000 

Baseline  plus  full  AV-1 

.538 

0.000 

Baseline  plus  OV-3 

.538 

0.000 

Baseline  plus  SV-5 

.538 

0.000 

Random  VDEA  Score  1 

.454 

-0.084 

4.3  Sensitivity  Analysis 

The  Hierarchy  Builder©  software  includes  a  sensitivity  analysis  tool  which  examines  the 
weight  of  each  value  and  measure  in  the  hierarchy.  This  sensitivity  analysis  tool  allows  the  user 
to  see  how  the  VDEA  score  would  be  changed  by  adjusting  the  weight  of  a  particular  value  or 
measure.  By  examining  how  weights  affect  the  scoring  of  alternatives,  the  decision-maker  can 
gain  insight  as  to  how  the  weighting  of  a  certain  value  affects  the  decision. 

4.3.1  Global  Weight  Sensitivity  Analysis 

The  Hierarchy  Builder©  software’s  sensitivity  analysis  tool  is  capable  of  perfonning 
sensitivity  analyses  both  globally  and  locally.  This  sensitivity  analysis  shows  the  decision  maker 
how  the  final  decision  may  be  affected  by  altering  the  weight  of  a  particular  value.  Since  the 
weights  were  chosen  and  validated  by  the  decision-maker,  the  current  weights  are  assumed 


130 


correct.  Sensitivity  analysis  is  provided  to  demonstrate  alternative  scenarios  and  allow  for 
further  verification  of  weight. 

4.3. 1.1  Alternative  Sensitivity  Analysis 

Sensitivity  analysis  in  a  multiple  alternative  situation  is  generally  more  useful  to  the 
decision-maker.  In  a  multiple  alternative  situation,  adjusting  the  weights  of  certain  values  or 
measures  may  lead  to  one  alternative  being  chosen  over  another.  In  some  cases,  a  small  change 
in  weight  may  lead  to  a  major  change  in  the  ranking  of  alternatives.  It  is  therefore  important  to 
closely  examine  the  sensitivity  curves  for  each  of  the  alternatives  to  ensure  that  the  weighting  is 
correct  and  the  proper  alternative  is  being  selected.  In  the  case  of  the  JFPASS,  the  alteration  of 
some  weights  may  affect  which  product  should  be  included  next  in  the  architecture.  With  more 
alternatives,  the  complexity  of  the  analysis  and  decision  increases,  and  minor  alterations  may 
have  a  larger  effect  on  the  outcome.  Any  changes  to  weights  must  be  well  vetted  through  the 
decision-maker  to  ensure  the  change  is  being  made  for  the  proper  reasons. 

4.3. 1.1.1  Alternatives  System  Effectiveness  Sensitivity  Analysis 

When  a  sensitivity  analysis  is  performed  on  all  alternatives,  the  best  performing  system 
is,  quite  obviously,  the  Perfect  system  alternative.  This  provides  a  good  point  of  comparison  for 
the  other  alternatives.  The  analysis  shows  that  if  the  local  weight  of  System  Effectiveness  is 
increased  to  one,  the  perfect  system  is  by  far  the  best  performing  alternative;  this  is  followed  by 
the  SV-7,  SV-8,  and  SV-9  alternative.  The  next  best  perfonning  alternative  is  the  Current 
system  plus  AV-1,  SV-7,  and  SV-5  included.  Though  this  is  the  next  best  performing 
alternative,  there  is  still  a  great  deal  of  value  not  being  earned.  Each  of  the  alternatives  has  a 
negative  slope  as  the  weight  of  System  Effectiveness  approaches  one.  This  chart  shows  the 
decision  maker  that  even  in  the  case  of  the  best  generated  alternative,  there  are  value  gaps.  In 
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addition,  if  the  global  weight  of  System  Effectiveness  is  altered  between  zero  and  one,  there  will 
be  no  effect  to  the  alternative  preference.  Figure  4.7  shows  the  sensitivity  analysis  for  System 
Effectiveness  with  all  alternatives  included. 


Figure  4.7  System  Effectiveness  Alternative  Sensitivity  Analysis 


4.3. 1.1.2  Alternatives  Capability  Sensitivity  Analysis 

When  the  Capability  value  is  examined,  there  are  significant  changes  to  the  alternative 
ranking  that  can  be  affected  by  changing  the  weight  of  the  Capability  value.  The  Perfect  system 
will  of  course  continue  to  perform  the  best  of  all  alternatives  as  the  weight  increases.  As  the 
global  weight  of  Capability  approaches  approximately  0.4,  the  Random  VDEA  Score  4 
alternative  begins  to  outperfonn  the  AV-1,  SV-7,  and  SV-5  scenario.  As  the  global  weight  of 
Capability  increases  or  decreases,  it  may  affect  the  decision  that  will  be  made.  Depending  on  the 


132 


alternative  chosen,  the  amount  of  adjustment  before  the  decision  is  impacted  alters.  In  the  case 
of  Capability,  if  the  weight  is  incorrect,  it  will  not  change  the  decision  from  this  point.  These 
rankings,  as  well  as  the  points  at  which  the  alternatives  change  order  can  be  seen  in  Figure  4.8. 
If  the  weight  is  set  to  any  value  between  zero  and  one,  a  new  ranking  can  be  found  by  observing 
the  order  of  alternatives.  Locally,  there  are  similar  changes  to  the  rankings  of  alternatives  when 
examined. 


Sensitivity  Analysis  for  Capability 


0  0.1  0.2  0.3  0.4  0.5  0.6  0.7  0.8  0.9  1 


♦'  Baseline 

■  JFPASS  Perfect  System 
A-  -Baseline  plus  SV-7 
•<  Baseline  plus  Full  AV-1 
■K—  Baseline  plus  OV-3 
9  Baseline  plus  OV-5 
+  Baseline  plus  SV-8 

- baseline  plus  SV-9 

-  Baseline  plus  AV-1  and  SV-5 
♦  Baseline  plus  AV-1,  SV-7,  and  SV-5 
Baseline  plus  SV-7,  SV-8,  and  SV-9 
—A— Random  VDEA  Score  1 
Random  VDEA  Score  2 
— —Random  VDEA  Score  3 
Random  VDEA  Score  4 


Figure  4.8  Capability  Alternative  Global  Sensitivity  Analysis 


4.3. 1.1.3  Alternatives  Maintainability  Sensitivity  Analysis 

As  the  global  weight  of  Maintainability  increases,  the  ranking  of  alternatives  changes 
significantly.  Several  alternatives’  values  drop  to  below  0.1  in  fact.  The  range  of  alternative 
scores  varies  between  0.05  and  0.82  (without  consideration  of  the  Perfect  System).  Capability 
however  only  varies  between  0.4  and  0.78.  This  shows  that  the  current  scoring  of 
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Maintainability  offers  a  great  deal  of  opportunities  for  improvement.  It  is  also  evident  that  in 
some  alternatives,  particularly  in  the  mid  0.6  value  range,  the  decision  is  very  sensitive  to  the 
weighting  of  Maintainability.  The  alternatives  in  which  the  Maintainability  measures  have 
attained  their  maximum  value  perfonn  much  better  in  a  Maintainability  sensitivity  analysis.  The 
random  alternatives  are  the  only  alternatives  that  score  differently  than  the  generated 
alternatives.  All  other  generated  alternative  either  rank  in  the  lower  or  higher  group,  due  to  the 
effect  of  the  Maintainability  measures.  The  generated  alternatives  generally  perform  together, 
since  the  addition  of  the  SV-7  product  allows  all  Maintainability  measures  to  maximize.  The 
random  alternatives  are  not  bound  by  all  Maintainability  measures  performing  together; 
therefore,  they  score  differently.  Locally,  the  alternatives  perfonn  similarly  to  globally,  but  the 
range  of  values  decreases.  Figure  4.9  shows  the  global  sensitivity  analysis  for  Maintainability. 


Sensitivity  Analysis  for  Maintainability 
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■■■*■■  Baseline  plus  OV-3 
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Random  VDEA  Score  2 
— t—  Random  VDEA  Score  3 
Random  VDEA  Score  4 
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Figure  4.9  Maintainability  Alternative  Global  Sensitivity  Analysis 
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4.3. 1.1.4  Alternatives  Interoperability  Sensitivity  Analysis 

The  sensitivity  analysis  for  Interoperability  shows  that  again  the  random  alternatives  are 
the  only  one  that  performs  significantly  differently.  All  other  alternatives  converge  to  the  same 
value  of  approximately  0.6  as  their  global  weights  approach  one.  As  the  weights  approach  zero, 
their  values  diverge  slightly  from  their  current  state,  but  the  rankings  do  not  change.  This 
implies  that  in  order  to  affect  any  significant  change  to  the  final  score,  the  Interoperability 
measures  must  score  differently  than  they  do  in  the  majority  of  the  alternatives.  In  this  case,  the 
weight  of  Interoperability  can  have  a  major  effect  on  the  decision.  If  the  weight  of 
Interoperability  is  increased  to  0.5,  the  SV-5  is  the  next  best  choice.  Until  that  point,  the 
decision  remains  unaltered.  In  an  analysis  of  the  scoring  of  each  alternative,  it  is  apparent  that 
each  of  the  random  alternatives  has  differing  scores  for  the  Interoperability  measures,  causing 
their  changes  in  the  sensitivity  analysis.  Figure  4.10  shows  the  global  sensitivity  analysis  for  the 
Interoperability  value. 


Sensitivity  Analysis  for  Interoperability 
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Figure  4.10  Interoperability  Alternative  Global  Sensitivity  Analysis 
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Chapter  5.  Conclusions  and  Recommendations 


This  chapter  provides  an  overview  of  the  research  completed  and  the  results  of  the 
JFPASS  system  analysis.  The  research  questions  for  this  effort  were  answered  by  leveraging  the 
Value-Focused  Thinking  (VFT)  process.  Recommendations  for  the  existing  architecture  as  well 
as  future  architecture  developments  in  the  Joint  Force  Protection  Advance  Security  System 
(JFPASS)  project  were  also  determined.  In  addition,  suggestions  for  improvement  of  the  Value- 
Driven  Enterprise  Architecture  (VDEA)  tool  are  discussed. 

5.1  Evaluation  Result 

The  research  questions  for  this  effort  were  “(1)  Can  the  VFT  process  be  applied  to  an 
evaluation  of  a  set  of  architectural  products?  (2)  What  is  the  resulting  Hierarchy  to  evaluate  a 
force  protection  system?  (3)  What  are  the  related  weights  and  measures  for  the  hierarchy?  (4) 
What  score  does  the  provided  architecture  score  based  on  this  hierarchy  and  where  are  the 
shortfalls  and  potential  areas  of  improvement?”  Each  of  these  questions  were  answered  during 
this  effort. 

It  was  shown  that  the  VFT  process  could  be  applied  to  evaluate  a  set  of  architectural 
products.  Through  research  into  existing  guidance  and  documentation  and  interviews  with 
decision-makers,  it  is  possible  to  detennine  what  values  are  important  to  those  decision-makers 
in  architectural  design.  Through  the  use  of  the  VFT  process,  a  Value-Driven  Enterprise 
Architecture  methodology  was  found.  Even  with  a  single  alternative,  the  VFT  process  was  able 
to  output  a  score  for  the  system  as  a  whole  to  be  used  as  a  baseline  for  future  improvements.  In 
addition  to  giving  the  decision-maker  a  baseline,  the  VFT  process  allowed  for  the  creation  of 
alternatives  that  could  show  the  possible  future  maturation  and  development  of  the  project. 
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These  alternatives  gave  the  decision-maker  a  set  of  comparison  criteria  to  determine  the  future 
direction  of  project  development. 


The  resulting  hierarchy  for  JFPASS  evaluation  was  developed  in  two  major  branches:  the 
System  Effectiveness  Branch  and  the  Architecture  Quality  branch.  This  effort  detennined  a 
possible  hierarchy  for  System  Effectiveness  evaluation.  This  Hierarchy  is  shown  in  Figure  5.1. 


Figure  5.1  System  Effectiveness  Hierarchy 


In  addition  to  the  hierarchy,  weights  were  assigned  to  each  value  and  measures  of 
effectiveness  assigned  to  each  lowest  tier  value.  The  weights  assigned  to  each  value  were 
determined  through  both  research  of  the  guidance  and  documentation  and  interviews  with  subject 
matter  experts  (SMEs)  in  the  field  of  force  protection  (FP).  The  resulting  16  measures  allow  the 
decision-maker  to  determine  the  effectiveness  of  the  system  in  question. 

Finally,  the  JFPASS  was  assigned  a  score  of  0.538  of  1.000  possible  value  units  through 
a  deterministic  analysis.  This  score  includes  the  value  earned  by  both  the  System  Effectiveness 
and  the  Architecture  Quality.  This  is  also  an  “earned  value”  as  opposed  to  a  final  score.  At  this 
point  in  time,  the  JFPASS  has  earned  0.538  of  its  possible  value  units.  As  it  is  still  early  in  its 
development,  there  is  time  to  earn  additional  value  for  this  system  and  improve  the  score.  This 
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score  also  serves  to  show  the  level  of  development  of  the  current  system,  as  well  as  which  areas 
are  lacking.  The  shortfalls  and  suggested  areas  of  improvement  are  presented  in  the  next  section. 

5.2  Recommendations 

Through  deterministic  analysis  and  sensitivity  analysis  of  the  system,  several 
recommendations  were  generated,  both  for  improvement  of  the  current  system  and  for  future 
development  of  the  system.  These  recommendations  are  intended  to  help  guide  work  on  the 
JFPASS  project  with  the  final  system  value  in  mind.  Final  determination  of  the  course  of  future 
system  development  lies  in  the  hands  of  the  decision-maker  and  system  sponsor,  but  the  scores 
and  sensitivity  analysis  provide  a  justification  for  possible  changes  to  system  development. 

5.2.1  Future  View  Development 

Several  areas  of  evaluation  in  System  Effectiveness  require  more  infonnation  or 
additional  views  to  properly  score.  The  addition  of  these  views  would  allow  for  the  scoring  of 
certain  measures  of  effectiveness,  providing  more  value  to  the  overall  system.  Through  a 
sensitivity  analysis  of  the  current  measures,  it  was  possible  to  detennine  the  maximum  benefit 
provided  by  each  measure  and  view.  Table  5.1  shows  the  maximum  benefit  that  the  creation  of 
each  new  view  would  have  on  the  total  score. 

The  JFPASS  Perfect  System  alternative  exists  for  decision-maker  comparison,  not  as  a 
practical  alternative.  The  highest  scoring  realistic  alternative  is  one  which  incorporates  a 
complete  SV-7,  SV-8,  and  SV-9.  The  next  alternative  in  the  ranking  requires  three  additional 
views,  each  of  which  builds  on  the  infonnation  in  others.  By  creating  these  additional  views  in  a 
specific  order,  value  can  be  added  more  quickly.  For  these  recommendations,  the  random 
alternatives  are  not  considered.  They  are  included  in  the  analysis  as  a  basis  of  comparison  for 
the  decision-maker.  These  recommendations  may  be  applied  to  cost-benefit  analysis. 
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Table  5.1  Maximum  Value  Benefit  of  New  Views 


Alternative  Name 

Score 

Maximum  Value 
Benefit 

JFPASS  Perfect  System 

0.890 

0.352 

Current  System  plus  AV-1,  SV-7,  and  SV-5 

0.704 

0.166 

JFPASS  Random  4 

0.686 

0.0148 

Current  System  plus  SV-7 

0.663 

0.0125 

JFPASS  Random  3 

0.583 

0.045 

Current  System  plus  AV-1  and  SV-5 

0.578 

0.040 

Current  System  plus  SV-8 

0.565 

0.027 

Current  System  plus  SV-9 

0.557 

0.019 

JFPASS  Random  2 

0.540 

0.002 

Baseline 

0.538 

0.000 

Current  System  plus  Full  AV-1 

0.538 

0.000 

Current  System  plus  OV-3 

0.538 

0.000 

Current  System  plus  SV-5 

0.538 

0.000 

JFPASS  Random  1 

0.454 

-0.084 

Based  on  this  analysis,  the  next  most  beneficial  product  to  create  would  be  the  SV-7, 
System  Performance  Parameters  Matrix.  This  view  allows  the  architect  to  add  infonnation 
regarding  the  parameters  to  which  system  components  are  designed.  This  includes  three  of  the 
four  Maintainability  measures  required  for  this  evaluation.  The  information  contained  in  the 
SV-7  product  may  already  be  used  in  the  system  design,  but  without  the  SV-7,  there  is  no  other 
place  in  the  architecture  that  the  information  can  be  found.  For  the  system  to  be  properly 
designed,  it  must  have  some  parameters  by  which  system  components  are  designed,  specifically 
regarding  their  supportability,  reliability,  and  recoverability.  These  ideas  must  simply  be 
included  in  the  architecture  to  ensure  compliance  with  the  parameters  and  proper  representation 
to  the  reader.  By  simply  adding  the  SV-7  product,  the  total  system  score  has  the  potential  to 
increase  by  0. 125  value  units  (if  all  required  information  is  included). 

Following  the  creation  of  the  SV-7,  an  SV-5,  Operational  Activity  to  Systems  Function 
view  would  create  the  next  highest  value  benefit.  The  SV-5  is  intended  to  show  the  reader  which 
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components  are  accomplishing  which  functions.  By  tracing  those  functions  up  to  operational 
requirements,  it  is  possible  to  determine  how  the  operational  needs  of  the  system  are  being  met, 
i.e.  using  which  system  components.  For  the  SV-5  to  be  of  any  benefit,  though,  the  AV-1  must 
also  be  updated  with  an  explicit  list  of  the  operational  requirements  for  the  system.  The  AV-1 
currently  includes  a  discussion  of  the  purpose  of  the  system,  but  lacks  any  specific  discussion  of 
the  problems  that  the  system  will  solve  and  the  constraints  by  which  the  system  is  being 
designed.  It  was  of  great  importance  to  the  decision-maker  that  the  system  accomplishes  its 
goals.  Therefore,  those  goals  must  be  outlined  explicitly.  The  AV-1  provides  the  best  context 
for  this  discussion.  These  details  may  also  be  included  in  some  appendix  to  the  architecture,  but 
should  be  included  with  the  architectural  package.  Without  a  list  of  operational  needs  or 
requirements,  the  SV-5  is  of  no  benefit  to  the  system.  If  the  SV-5/AV-1  update  is  completed 
following  the  creation  of  the  SV-7,  the  three  updates  have  a  positive  cumulative  effect  on  the 
system  score.  They  will  cause  a  positive  change  of  0.166  value  units  to  the  final  score. 

The  next  most  beneficial  view  would  be  the  SV-8,  Systems  Evolution  Description.  The 
inclusion  of  the  SV-8  allows  the  architecture  evaluator  the  ability  to  score  the  ADAPTION 
measure.  There  are  currently  no  details  included  in  the  architecture  regarding  the  Flexibility  of 
the  system.  The  SV-8  has  the  ability  to  show  the  reader  how  the  system  may  evolve  not  only 
further  into  the  acquisition  and  design  process,  but  under  operational  constraints.  The  inclusion 
of  an  SV-8  with  the  necessary  information  to  score  Adaptability  may  cause  a  benefit  of  0.027 
value  units. 

Finally,  the  SV-9,  Systems  Technology  Forecast  view  would  provide  the  next  most 
benefit  to  the  system  score.  The  inclusion  of  the  SV-9  has  the  ability  to  add  1.9  value  units  to  the 
total  score.  The  SV-9  product  allows  the  reader  to  determine  the  current  Technology  Readiness 
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Level  (TRL)  of  the  components  included  in  the  architecture.  The  TRL  of  each  component  is 
required  to  determine  the  overall  system  TRL  for  the  TECHNOLOGICAL  AVAILABILITY  measure. 

The  OV-3,  while  included  as  supporting  documentation  for  some  measures,  is  not 
required  for  this  system.  It  is  possible  to  determine  the  infonnation  included  in  the  OV-3 
without  its  inclusion  for  scoring.  The  OV-3  may  include  other  information  of  use  to  the  architect 
or  decision-maker,  therefore  its  inclusion  should  not  be  complete  discounted.  The  architect  is 
also  bound  by  the  milestone  decision  point  requirements  for  architecture  products.  In  other  force 
protection  systems,  this  view  may  be  required  to  score  the  architecture.  The  addition  of  an  SV-5 
alone  will  not  add  any  value  either,  unless  it  is  added  with  the  AV-1  updates.  Conversely,  the 
addition  of  a  complete  AV-1  will  not  have  any  positive  effect  on  the  system  score  without  the 
SV-5. 

5.2.2  System  Strengthening 

In  addition  to  future  development  of  views,  several  steps  may  be  taken  to  strengthen  the 
current  system  and  its  score.  In  some  cases,  more  value  may  be  earned  by  improving  upon 
information  already  included  or  updating  design  decisions  based  on  the  decision-maker’s  most 
important  values.  In  other  cases,  a  measure  may  already  score  full  value,  but  the  inclusion  of 
additional  infonnation  may  make  the  architecture  more  easily  scored  and  read. 

The  THREAT  DETECTION,  THREAT  ASSESSMENT  and  WARNING  PLAN  measures  were  all 
scored  positively.  Though  they  were  scored  at  their  highest  level,  it  is  possible  to  make  them 
more  easily  accessible  to  architecture  readers.  Each  of  these  measures  refers  to  the  inclusion  of  a 
plan  related  to  the  measure.  To  determine  the  degree  of  attainment  of  these  measures,  the  OV-5 
product  was  used.  Since  the  activities  required  to  accomplish  each  of  these  concepts  were 
included,  they  were  scored  positively.  Including  actual  text  versions  of  the  plans  may  assist  not 
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only  the  architecture  readers,  but  also  future  users  of  the  architecture.  Each  installation  is 
required,  regardless  of  service  component  to  have  official,  written  plans  for  these  concepts.  The 
inclusion  of  skeleton  versions  of  these  plans  in  the  architecture  would  assist  in  the  scoring  and 
ensure  compliance  of  each  installation. 

The  ENVIRONMENTAL  IMPACT  measure  was  scored  “CONUS  and  Contingency 
constratints.”  This  measure  is  capable  of  earning  0.2  additional  value  units  by  adding  Host 
Nation  constraints  to  the  current  consideration.  Including  international  environmental  policy 
documents  in  the  TV-1  would  allow  this  measure  to  be  scored  at  a  higher  level.  It  may  also  be 
possible  to  include  the  environmental  policies  of  several  nations  that  the  U.S.  military 
historically  operates  and  has  standing  military  commitments  (or  Status  of  Forces  Agreements 
(SOFA))  with. 

The  MONETARY  PRACTICALITY  measures  refer  to  the  budgetary  constraints  that  all 
government  programs  are  subject  to.  The  ability  to  construct  a  system  within  budget  is  of  major 
concern  to  all  stakeholders  in  a  project.  The  inclusion  of  specific  cost  information  to  the 
architecture  would  not  only  allow  the  scoring  of  these  measures,  but  would  add  verification  to 
the  stakeholders  of  a  system’s  fiscal  viability.  A  total  system  estimate  and  program  budget  must 
of  course  be  included  as  well  in  order  to  compare  the  cost  information  to.  The  OV-5  product  has 
the  ability  to  display  this  information.  Initial  cost  and  life  cycle  costs  may  also  be  included  in  the 
AV-1  product,  although  the  costs  would  not  be  itemized  by  component. 

The  two  Net-Centric  Enterprise  Solutions  for  Interoperability  (NESI)  measures  also  have 
room  for  improvement.  Currently,  the  NESI  DEVELOPMENT  measure  is  scored  positively,  but  its 
assessment  may  be  improved  by  explicitly  including  the  NESI  documents  in  the  TV-1.  The 
assessment  for  this  measure  was  done  by  comparing  the  included  system  design  and  design 
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documents  to  the  NESI  guidance  and  determining  that  the  system  was  being  constructed  with 
Net-Centricity  in  mind.  Including  the  NESI  documents  in  the  architecture  would  show  the 
architecture  reader  that  the  system  was  in  fact  designed  with  these  concepts  in  mind.  In  addition 
to  including  the  Net-Centricity  documents,  a  NESI  evaluation  should  also  be  completed  on  the 
architecture.  Simply  completing  the  evaluation  would  allow  for  a  positive  score  of  the  NESI 
EVALUATION  measure,  but  inclusion  of  the  evaluation  in  the  architecture,  as  well  as  a  positive 
score  would  assist  in  the  scoring  of  the  architecture. 

5.3  Model  Strengths 

The  creation  of  the  Value  Driven  Enterprise  Architecture  tool  allows  a  force  protection 
architecture  to  be  objectively  evaluated.  This  tool  gives  the  decision-maker  an  objective 
numerical  score  from  which  to  base  future  revisions  and  additions  to  the  architecture.  This 
baseline  combined  with  analysis  of  the  score  shows  the  most  beneficial  views  to  be  created  in  the 
future  and  improvements  that  may  be  made  to  the  existing  architecture. 

In  interviews  with  several  force  protection  experts,  the  system  was  found  to  be  all 
inclusive  of  the  important  values  for  force  protection.  Each  of  the  SMEs  found  no  major  areas  of 
force  protection  that  were  not  included  in  the  values  of  this  model.  By  creating  a  system  built 
around  the  values  alone,  a  comprehensive  force  protection  system  may  be  constructed. 

This  model  allows  a  project  stakeholder  or  sponsor  to  “score”  a  system  based  solely  on 
its  architecture.  This  is  useful  since  many  acquisition  decisions  are  made  based  solely  on 
architectural  products.  Having  a  tool  to  evaluate  them  allow  for  objective  evaluation  of  the 
architecture. 
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5.4  Model  Weaknesses 


While  the  model  is  useful  for  the  decision-maker,  there  are  limitations  and  several  areas 
that  may  be  improved.  The  extensibility  of  the  measures  under  the  System  Effectiveness  branch 
may  be  limited.  The  measures  as  presented  here  were  useful  for  a  force  protection  system  in  the 
very  early  stages  of  development.  As  the  system  matures,  new  measures  will  need  to  be  created 
to  keep  up  with  the  changing  needs  of  the  system.  At  some  point,  it  may  be  necessary  to 
improve  the  granularity  and  objectivity  of  measures  to  better  evaluate  increased  complexity. 

Another  weakness  of  this  model  is  associated  with  the  inherent  uncertainty  of  a  VFT 
model.  The  Single  Dimension  Value  Functions  (SDVFs)  and  weights  are  based  on  input  from 
the  decision-maker  and  subject  matter  experts.  They  were  constructed  to  match  the  values  of  the 
people  involved  in  their  construction,  but  in  the  end  these  instruments  are  only  the  opinion  of 
those  who  were  involved  in  their  creation.  There  are  other  possible  combinations  of  weights  and 
SDVFs  which  may  also  measure  the  system. 

There  remains  a  certain  level  of  ambiguity  and  subjectivity  involved  in  the  scoring  of  the 
architecture.  Though  the  scores  were  reached  by  consensus  and  taken  directly  from  the 
architectural  products,  some  scores  may  not  be  accurate.  The  descriptions  included  in  Chapter  3 
allow  for  repeatability,  but  some  subjective  decisions  must  still  be  made  regarding  the  scores. 

5.5  Future  Research 

This  effort  has  opened  the  door  for  several  additional  areas  of  research.  The  research 
may  be  extended  to  refine  values,  measures,  and  SDVFs  and  update  the  hierarchy  to  include 
future  assessments  of  the  same  system.  The  value  hierarchy  derived  in  this  study  may  be  applied 
to  other  force  protection  systems.  Individual  projects  outside  the  scope  of  JFPASS  may  also  use 
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the  methodology  to  objectively  score  their  viability.  Component  selection  may  also  be 
influenced  by  the  value  hierarchy  or  the  VDEA  score. 

The  JFPASS  system  is  currently  in  an  early  stage  of  development  and  will  continue  to 
mature.  As  the  project  grows,  some  measures  may  be  revised  to  reflect  the  updated  system. 
Several  measures  currently  detennine  the  existence  of  certain  concepts,  but  in  the  future,  they 
may  be  used  to  measure  quality  of  the  achievement  of  these  values. 

The  hierarchy  derived  in  this  study  includes  all  of  the  major  values  associated  with  any 
force  protection  system  and  perhaps  different  types  of  systems.  The  hierarchy  may  be  applied  to 
FP  systems  outside  the  JFPASS  to  complete  similar  evaluations.  It  may  also  be  used  to  design 
future  FP  systems  in  order  to  ensure  their  compliance  with  the  most  important  aspects  of  the 
force  protection. 

The  JFPASS  system  will  have  several  individual  projects  created  under  its  “umbrella.” 
As  they  projects  emerge,  they  may  also  be  scored  using  the  same  model.  The  System 
Effectiveness  branch  will  allow  for  the  evaluation  of  any  force  protection  system,  particularly 
those  within  the  purview  of  JFPASS.  Outside  the  context  of  JFPASS,  other  Force  protection 
projects  may  also  be  compared  using  this  architecture.  As  projects  are  submitted  to  the  JFPASS 
office,  they  may  be  compared  using  this  architecture.  The  hierarchy  allows  for  project  selection 
from  a  number  of  alternatives  in  addition  to  its  ability  to  generate  new  alternatives. 
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Appendix  A.  Ilities  Master  List 


accessibility 
accountability 
accuracy 
adaptability 
administrability 
affordability 
agility 
applicability 
auditability 
availability 
capability 
changeability 
communication 
compatibility 
complexity 
compliancy 
composability 
configurability 
consistency 
constructability 
controllability 
credibility 
customizability 
data  integrity 
degradability 
demonstrability 
dependability 
deployability 
diagnoseability 
distributability 
durability 
effectiveness 
efficiency 


evolvability 

extensibility 

feasibility 

_ fidelity _ 

flexibility 

functionality 

installability 

interchangeability 

internationalizability 

interoperability 

learnability 

maintainability 

manageability 

mobility 

modifiability 

modularity 

nomadicity 

_ openness _ 

operability 

performance 

personalizability 

portability 

practicality 

_ precision _ 

predictability 

produceability 

protectability 

purposefulness 

_ quality _ 

readability 

recoverability 

relevance 

reliability 


repairability 

repeatability 

reproducibility 

resiliency 

responsiveness 

reusability 

_ robustness _ 

scalability 

seamlessness 

securability 

_ security _ 

serviceability 

Simplicity 

Stability 

stakeholder  involvement 
subscribability 

_ supportability _ 

survivability 

_ susceptability _ 

sustainability 

tailorability 

testability 

timeliness 

traceability 

trainability 

transactionality 

understandability 

upgradeability 

usability 

_ utility _ 

versatility 

vulnerability 
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Appendix  B.  System  Effectiveness  Groups  and  Synonyms 
Group  1:  Capability 

Subgroup  1:  Purposefulness 

Synonyms:  Relevance,  Applicability,  Utility,  Perfonnance,  Robustness, 
Functionality 
Subgroup  2:  Practicality 

Synonyms:  Deployability,  Affordability,  Produceability,  Constructability, 
Efficiency,  Feasibility,  Installability,  Operability 
Subgroup  3:  Flexibility 

Synonyms:  Modularity,  Responsiveness,  Configurability,  Versatility, 
Adaptability,  Mobility,  Agility 

Group  2:  Maintainability 

Subgroup  1:  Dependability 

Subgroup  1.1:  Supportability 

Synonyms:  Repairability,  Sustainability,  Serviceability,  Maintainability 
Subgroup  1.2:  Reliability 

Synonyms:  Dependability,  Degradability,  Fidelity,  Stability 
Subgroup  2:  Resiliency 

Subgroup  2. 1 :  Survivability 

Synonyms:  Susceptibility 
Subgroup  2.2:  Recoverability 

Synonyms:  Diagnosability 
Group  3:  Interoperability 

Subgroup  1:  Communication 
Subgroup  2:  Interchangeability 

Synonyms:  Compatibility,  Internationalizability 
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Appendix  C.  Supplemental  Deterministic  Analysis  Charts 


JFPASS  17  February  2009  0.538 


■  NESI  Development 

■  NESI  Evaluation 

■  Reliability  Requirements 

■  Operational  Needs 

□  Threat  Detection 

□  Threat  Assessment 

□  Warning  Plan 

■  Redundancy 

□  Supportability  Requirements 

□  Access  Control 

□  Document  Protection 

□  Joint  Operations 

EJ  DoDAF  Compliancy 

■  OV  Readability 

□  SV  Readability 

□  Adaptation 

□  Recoverability  Requirements 

□  Scale 

□  Decomposition 

■  Access 

Figure  C.l  JFPASS  Score  Fundamental  Objective  -  Measures 
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JFPASS  17  February  2009  0.538 


□  System  Effectiveness  □  Architecture  Quality 


Figure  C.2  JFPASS  Score  Fundamental  Objective  -  Values 
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JFPASS  17  February  2009  0.413 


□  NESI  Development 

□  Reliability  Requirements 

□  Threat  Detection 

□  Warning  Plan 

□  Supportability  Requirements 

□  Adaptation 

□  Technological  Availability 

□  Monetary  Practicality  -  Initial 


■  NESI  Evaluation 

■  Operational  Needs 

□  Threat  Assessment 

□  Redundancy 

□  Joint  Operations 

□  Recoverability  Requirements 

□  Environmental  Impact 

□  Monetary  Practicality  -  Maintenance 


Figure  C.3  System  Effectiveness  Score  -  Measures 
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JFPASS  17  February  2009  0.514 


□  Operational  Needs 

□  Threat  Assessment 

□  Adaptation 

□  Environmental  Impact 

□  Monetary  Practicality  -  Maintenance 


■  Threat  Detection 

■  Warning  Plan 

□  Technological  Availability 

□  Monetary  Practicality  -  Initial 


Figure  C.4  Capability  Score  -  Measures 
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JFPASS  17  February  2009  0.514 


□  Purposefulness  E  Practicality  □Flexibility 

Figure  C.5  Capability  Score  -  Values 
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JFPASS  17  February  2009  0.750 


□  Operational  Needs  DThreat  Detection  □  Threat  Assessment 


Figure  C.6  Purposefulness  Score  -  Measures 


□  WarningPlan 
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□  Technological  Availability  □  Environmental  Impact 

□  Monetary  Practicality  -  Initial  □  Monetary  Practicality  -  Maintenance 

Figure  C.7  Practicality  Score  -  Measures 
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JFPASS  17  February  2009  0.060 


□  Reliability  Requirements  ■  Redundancy  dSupportability  Requirements  □  Recoverability  Requirements 


Figure  C.8  Maintainability  Score  -  Measures 
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JFPASS  17  February  2009  0.060 


□  Dependability  □  Resiliency 


Figure  C.9  Maintainability  Score  -  Values 
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JFPASS  17  February  2009  0.600 


□  NESI  Development  □  NESI  Evaluation  □  Joint  Operations 


Figure  C.10  Interoperability  Score  -  Measures 
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JFPASS  17  February  2009  0.600 


□  Communication  □  Interchangeability 


Figure  C.ll  Interoperability  Score  -  Values 
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Appendix  D.  Alternative  Scores 


Alternative  Name:  Baseline 

Measure  Name 

Measurement 

OPERATIONAL  NEEDS 

0 

THREAT  DETECTION 

Yes 

THREAT  ASSESSMENT 

Yes 

WARNING  PLAN 

Yes 

TECHNOLOGICAL  AVAILABILITY 

TRL  1 

ENVIRONMENTAL  IMPACT 

CONUS  and  Contingency  constraints 

MONETARY  PRACTICALITY  - 
INITIAL 

Cost  Unknown 

MONETARY  PRACTICALITY  - 
MAINTENANCE 

Cost  unknown 

ADAPTATION 

Static 

JOINT  OPERATIONS 

Yes 

NESI  DEVELOPMENT 

Yes 

NESI  EVALUATION 

No 

ACCESS 

Access  between  3  and  7  days 

PRODUCT  LOCAT ABILITY 

Less  than  5  min 

ACCESS  CONTROL 

Appropriate  protection  implemented 

DOCUMENT  PROTECTION 

Plan  exists,  all  products  controlled 

FILE  MANAGEMENT 

No  system 

FILE  FORMAT 

General  File  Format 

SCALE 

Most  views 

DECOMPOSITION 

3+  Levels 

TOOL  FORMAT 

All  views 

DODAF  COMPLIANCY 

0.83 

REQUIREMENT  TRACEABILITY 

0 

INTERNAL  CONSISTENCY 

0.83 

EXTERNAL  CONSISTENCY 

0.83 

SME  EFFECTIVENESS 

No  Plan 

SME  INVOLVEMENT 

No  involvement 

SUPPORT  ABILITY  REQUIREMENTS 

No 

RELIABILITY  REQUIREMENTS 

No 

REDUNDANCY 

Some  systems,  single  redundancy 

RECOVERABILITY 

REQUIREMENTS 

No 

CONNECTIONS 

0.83 

ARCHITECTURE  REDUNDANCY 

Between  0  and  1:500 

ARCHITECTURE  ECONOMY 

No 

OV  READABILITY 

0.8 

SV  READABILITY 

0.6 
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Alternative  Name:  JFPASS  Perfect  System 

Measure  Name 

Measurement 

OPERATIONAL  NEEDS 

1 

THREAT  DETECTION 

Yes 

THREAT  ASSESSMENT 

Yes 

WARNING  PLAN 

Yes 

TECHNOLOGICAL  AVAILABILITY 

TRL  9 

ENVIRONMENTAL  IMPACT 

CONUS,  Contingency,  and  Host  Nation  constraints 

MONETARY  PRACTICALITY  - 
INITIAL 

<95%  budget 

MONETARY  PRACTICALITY  - 
MAINTENANCE 

<95%  budget 

ADAPTATION 

Minimal  effort 

JOINT  OPERATIONS 

Yes 

NESI  DEVELOPMENT 

Yes 

NESI  EVALUATION 

Yes 

ACCESS 

Access  between  3  and  7  days 

PRODUCT  LOCAT ABILITY 

Less  than  5  min 

ACCESS  CONTROL 

Appropriate  protection  implemented 

DOCUMENT  PROTECTION 

Plan  exists,  all  products  controlled 

FILE  MANAGEMENT 

No  system 

FILE  FORMAT 

General  File  Format 

SCALE 

Most  views 

DECOMPOSITION 

3+  Levels 

TOOL  FORMAT 

All  views 

DODAF  COMPLIANCY 

0.83 

REQUIREMENT  TRACEABILITY 

0 

INTERNAL  CONSISTENCY 

0.83 

EXTERNAL  CONSISTENCY 

0.83 

SME  EFFECTIVENESS 

No  Plan 

SME  INVOLVEMENT 

No  involvement 

SUPPORT  ABILITY  REQUIREMENTS 

Yes 

RELIABILITY  REQUIREMENTS 

Yes 

REDUNDANCY 

All  systems,  multiple  redundancy 

RECOVERABILITY 

REQUIREMENTS 

Yes 

CONNECTIONS 

0.83 

ARCHITECTURE  REDUNDANCY 

Between  0  and  1:500 

ARCHITECTURE  ECONOMY 

No 

OV  READABILITY 

0.8 

SV  READABILITY 

0.6 
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Alternative  Name:  Current  System  plus  SV-7 

Measure  Name 

Measurement 

OPERATIONAL  NEEDS 

0 

THREAT  DETECTION 

Yes 

THREAT  ASSESSMENT 

Yes 

WARNING  PLAN 

Yes 

TECHNOLOGICAL  AVAILABILITY 

TRL  1 

ENVIRONMENTAL  IMPACT 

CONUS  and  Contingency  constraints 

MONETARY  PRACTICALITY  -  INITIAL 

Cost  Unknown 

MONETARY  PRACTICALITY  - 
MAINTENANCE 

Cost  unknown 

ADAPTATION 

Static 

JOINT  OPERATIONS 

Yes 

NESI  DEVELOPMENT 

Yes 

NESI  EVALUATION 

No 

ACCESS 

Access  between  3  and  7  days 

PRODUCT  LOCAT ABILITY 

Less  than  5  min 

ACCESS  CONTROL 

Appropriate  protection  implemented 

DOCUMENT  PROTECTION 

Plan  exists,  all  products  controlled 

FILE  MANAGEMENT 

No  system 

FILE  FORMAT 

General  File  Fonnat 

SCALE 

Most  views 

DECOMPOSITION 

3+  Levels 

TOOL  FORMAT 

All  views 

DODAF  COMPLIANCY 

0.83 

REQUIREMENT  TRACEABILITY 

0 

INTERNAL  CONSISTENCY 

0.83 

EXTERNAL  CONSISTENCY 

0.83 

SME  EFFECTIVENESS 

No  Plan 

SME  INVOLVEMENT 

No  involvement 

SUPPORT  ABILITY  REQUIREMENTS 

Yes 

RELIABILITY  REQUIREMENTS 

Yes 

REDUNDANCY 

Some  systems,  single  redundancy 

RECOVERABILITY  REQUIREMENTS 

Yes 

CONNECTIONS 

0.83 

ARCHITECTURE  REDUNDANCY 

Between  0  and  1:500 

ARCHITECTURE  ECONOMY 

No 

OV  READABILITY 

0.8 

SV  READABILITY 

0.6 
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Alternative  Name:  Current  System  plus  Full  AV-1 

Measure  Name 

Measurement 

OPERATIONAL  NEEDS 

0 

THREAT  DETECTION 

Yes 

THREAT  ASSESSMENT 

Yes 

WARNING  PLAN 

Yes 

TECHNOLOGICAL  AVAILABILITY 

TRL  1 

ENVIRONMENTAL  IMPACT 

CONUS  and  Contingency  constraints 

MONETARY  PRACTICALITY  -  INITIAL 

Cost  Unknown 

MONETARY  PRACTICALITY  - 
MAINTENANCE 

Cost  unknown 

ADAPTATION 

Static 

JOINT  OPERATIONS 

Yes 

NESI  DEVELOPMENT 

Yes 

NESI  EVALUATION 

No 

ACCESS 

Access  between  3  and  7  days 

PRODUCT  LOCAT ABILITY 

Less  than  5  min 

ACCESS  CONTROL 

Appropriate  protection  implemented 

DOCUMENT  PROTECTION 

Plan  exists,  all  products  controlled 

FILE  MANAGEMENT 

No  system 

FILE  FORMAT 

General  File  Fonnat 

SCALE 

Most  views 

DECOMPOSITION 

3+  Levels 

TOOL  FORMAT 

All  views 

DODAF  COMPLIANCY 

0.83 

REQUIREMENT  TRACEABILITY 

0 

INTERNAL  CONSISTENCY 

0.83 

EXTERNAL  CONSISTENCY 

0.83 

SME  EFFECTIVENESS 

No  Plan 

SME  INVOLVEMENT 

No  involvement 

SUPPORT  ABILITY  REQUIREMENTS 

No 

RELIABILITY  REQUIREMENTS 

No 

REDUNDANCY 

Some  systems,  single  redundancy 

RECOVERABILITY  REQUIREMENTS 

No 

CONNECTIONS 

0.83 

ARCHITECTURE  REDUNDANCY 

Between  0  and  1:500 

ARCHITECTURE  ECONOMY 

No 

OV  READABILITY 

0.8 

SV  READABILITY 

0.6 
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Alternative  Name:  Current  System  plus  OV-3 

Measure  Name 

Measurement 

OPERATIONAL  NEEDS 

0 

THREAT  DETECTION 

Yes 

THREAT  ASSESSMENT 

Yes 

WARNING  PLAN 

Yes 

TECHNOLOGICAL  AVAILABILITY 

TRL  1 

ENVIRONMENTAL  IMPACT 

CONUS  and  Contingency  constraints 

MONETARY  PRACTICALITY  -  INITIAL 

Cost  Unknown 

MONETARY  PRACTICALITY  - 
MAINTENANCE 

Cost  unknown 

ADAPTATION 

Static 

JOINT  OPERATIONS 

Yes 

NESI  DEVELOPMENT 

Yes 

NESI  EVALUATION 

No 

ACCESS 

Access  between  3  and  7  days 

PRODUCT  LOCAT ABILITY 

Less  than  5  min 

ACCESS  CONTROL 

Appropriate  protection  implemented 

DOCUMENT  PROTECTION 

Plan  exists,  all  products  controlled 

FILE  MANAGEMENT 

No  system 

FILE  FORMAT 

General  File  Fonnat 

SCALE 

Most  views 

DECOMPOSITION 

3+  Levels 

TOOL  FORMAT 

All  views 

DODAF  COMPLIANCY 

0.83 

REQUIREMENT  TRACEABILITY 

0 

INTERNAL  CONSISTENCY 

0.83 

EXTERNAL  CONSISTENCY 

0.83 

SME  EFFECTIVENESS 

No  Plan 

SME  INVOLVEMENT 

No  involvement 

SUPPORT  ABILITY  REQUIREMENTS 

No 

RELIABILITY  REQUIREMENTS 

No 

REDUNDANCY 

Some  systems,  single  redundancy 

RECOVERABILITY  REQUIREMENTS 

No 

CONNECTIONS 

0.83 

ARCHITECTURE  REDUNDANCY 

Between  0  and  1:500 

ARCHITECTURE  ECONOMY 

No 

OV  READABILITY 

0.8 

SV  READABILITY 

0.6 
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Alternative  Name:  Current  System  plus  SV-5 

Measure  Name 

Measurement 

OPERATIONAL  NEEDS 

0 

THREAT  DETECTION 

Yes 

THREAT  ASSESSMENT 

Yes 

WARNING  PLAN 

Yes 

TECHNOLOGICAL  AVAILABILITY 

TRL  1 

ENVIRONMENTAL  IMPACT 

CONUS  and  Contingency  constraints 

MONETARY  PRACTICALITY  -  INITIAL 

Cost  Unknown 

MONETARY  PRACTICALITY  - 
MAINTENANCE 

Cost  unknown 

ADAPTATION 

Static 

JOINT  OPERATIONS 

Yes 

NESI  DEVELOPMENT 

Yes 

NESI  EVALUATION 

No 

ACCESS 

Access  between  3  and  7  days 

PRODUCT  LOCAT ABILITY 

Less  than  5  min 

ACCESS  CONTROL 

Appropriate  protection  implemented 

DOCUMENT  PROTECTION 

Plan  exists,  all  products  controlled 

FILE  MANAGEMENT 

No  system 

FILE  FORMAT 

General  File  Fonnat 

SCALE 

Most  views 

DECOMPOSITION 

3+  Levels 

TOOL  FORMAT 

All  views 

DODAF  COMPLIANCY 

0.83 

REQUIREMENT  TRACEABILITY 

0 

INTERNAL  CONSISTENCY 

0.83 

EXTERNAL  CONSISTENCY 

0.83 

SME  EFFECTIVENESS 

No  Plan 

SME  INVOLVEMENT 

No  involvement 

SUPPORT  ABILITY  REQUIREMENTS 

No 

RELIABILITY  REQUIREMENTS 

No 

REDUNDANCY 

Some  systems,  single  redundancy 

RECOVERABILITY  REQUIREMENTS 

No 

CONNECTIONS 

0.83 

ARCHITECTURE  REDUNDANCY 

Between  0  and  1:500 

ARCHITECTURE  ECONOMY 

No 

OV  READABILITY 

0.8 

SV  READABILITY 

0.6 

164 


Alternative  Name:  Current  System  plus  SV-8 

Measure  Name 

Measurement 

OPERATIONAL  NEEDS 

0 

THREAT  DETECTION 

Yes 

THREAT  ASSESSMENT 

Yes 

WARNING  PLAN 

Yes 

TECHNOLOGICAL  AVAILABILITY 

TRL  1 

ENVIRONMENTAL  IMPACT 

CONUS  and  Contingency  constraints 

MONETARY  PRACTICALITY  -  INITIAL 

Cost  Unknown 

MONETARY  PRACTICALITY  - 
MAINTENANCE 

Cost  unknown 

ADAPTATION 

Minimal  effort 

JOINT  OPERATIONS 

Yes 

NESI  DEVELOPMENT 

Yes 

NESI  EVALUATION 

No 

ACCESS 

Access  between  3  and  7  days 

PRODUCT  LOCAT ABILITY 

Less  than  5  min 

ACCESS  CONTROL 

Appropriate  protection  implemented 

DOCUMENT  PROTECTION 

Plan  exists,  all  products  controlled 

FILE  MANAGEMENT 

No  system 

FILE  FORMAT 

General  File  Fonnat 

SCALE 

Most  views 

DECOMPOSITION 

3+  Levels 

TOOL  FORMAT 

All  views 

DODAF  COMPLIANCY 

0.83 

REQUIREMENT  TRACEABILITY 

0 

INTERNAL  CONSISTENCY 

0.83 

EXTERNAL  CONSISTENCY 

0.83 

SME  EFFECTIVENESS 

No  Plan 

SME  INVOLVEMENT 

No  involvement 

SUPPORT  ABILITY  REQUIREMENTS 

No 

RELIABILITY  REQUIREMENTS 

No 

REDUNDANCY 

Some  systems,  single  redundancy 

RECOVERABILITY  REQUIREMENTS 

No 

CONNECTIONS 

0.83 

ARCHITECTURE  REDUNDANCY 

Between  0  and  1:500 

ARCHITECTURE  ECONOMY 

No 

OV  READABILITY 

0.8 

SV  READABILITY 

0.6 

165 


Alternative  Name:  Current  System  plus  SV-9 

Measure  Name 

Measurement 

OPERATIONAL  NEEDS 

0 

THREAT  DETECTION 

Yes 

THREAT  ASSESSMENT 

Yes 

WARNING  PLAN 

Yes 

TECHNOLOGICAL  AVAILABILITY 

TRL  9 

ENVIRONMENTAL  IMPACT 

CONUS  and  Contingency  constraints 

MONETARY  PRACTICALITY  -  INITIAL 

Cost  Unknown 

MONETARY  PRACTICALITY  - 
MAINTENANCE 

Cost  unknown 

ADAPTATION 

Static 

JOINT  OPERATIONS 

Yes 

NESI  DEVELOPMENT 

Yes 

NESI  EVALUATION 

No 

ACCESS 

Access  between  3  and  7  days 

PRODUCT  LOCAT ABILITY 

Less  than  5  min 

ACCESS  CONTROL 

Appropriate  protection  implemented 

DOCUMENT  PROTECTION 

Plan  exists,  all  products  controlled 

FILE  MANAGEMENT 

No  system 

FILE  FORMAT 

General  File  Fonnat 

SCALE 

Most  views 

DECOMPOSITION 

3+  Levels 

TOOL  FORMAT 

All  views 

DODAF  COMPLIANCY 

0.83 

REQUIREMENT  TRACEABILITY 

0 

INTERNAL  CONSISTENCY 

0.83 

EXTERNAL  CONSISTENCY 

0.83 

SME  EFFECTIVENESS 

No  Plan 

SME  INVOLVEMENT 

No  involvement 

SUPPORT  ABILITY  REQUIREMENTS 

No 

RELIABILITY  REQUIREMENTS 

No 

REDUNDANCY 

Some  systems,  single  redundancy 

RECOVERABILITY  REQUIREMENTS 

No 

CONNECTIONS 

0.83 

ARCHITECTURE  REDUNDANCY 

Between  0  and  1:500 

ARCHITECTURE  ECONOMY 

No 

OV  READABILITY 

0.8 

SV  READABILITY 

0.6 

166 


Alternative  Name:  Current  System  plus  AV-1  and  SV-5 

Measure  Name 

Measurement 

OPERATIONAL  NEEDS 

1 

THREAT  DETECTION 

Yes 

THREAT  ASSESSMENT 

Yes 

WARNING  PLAN 

Yes 

TECHNOLOGICAL  AVAILABILITY 

TRL  1 

ENVIRONMENTAL  IMPACT 

CONUS  and  Contingency  constraints 

MONETARY  PRACTICALITY  -  INITIAL 

Cost  Unknown 

MONETARY  PRACTICALITY  - 
MAINTENANCE 

Cost  unknown 

ADAPTATION 

Static 

JOINT  OPERATIONS 

Yes 

NESI  DEVELOPMENT 

Yes 

NESI  EVALUATION 

No 

ACCESS 

Access  between  3  and  7  days 

PRODUCT  LOCAT ABILITY 

Less  than  5  min 

ACCESS  CONTROL 

Appropriate  protection  implemented 

DOCUMENT  PROTECTION 

Plan  exists,  all  products  controlled 

FILE  MANAGEMENT 

No  system 

FILE  FORMAT 

General  File  Fonnat 

SCALE 

Most  views 

DECOMPOSITION 

3+  Levels 

TOOL  FORMAT 

All  views 

DODAF  COMPLIANCY 

0.83 

REQUIREMENT  TRACEABILITY 

0 

INTERNAL  CONSISTENCY 

0.83 

EXTERNAL  CONSISTENCY 

0.83 

SME  EFFECTIVENESS 

No  Plan 

SME  INVOLVEMENT 

No  involvement 

SUPPORT  ABILITY  REQUIREMENTS 

No 

RELIABILITY  REQUIREMENTS 

No 

REDUNDANCY 

Some  systems,  single  redundancy 

RECOVERABILITY  REQUIREMENTS 

No 

CONNECTIONS 

0.83 

ARCHITECTURE  REDUNDANCY 

Between  0  and  1:500 

ARCHITECTURE  ECONOMY 

No 

OV  READABILITY 

0.8 

SV  READABILITY 

0.6 
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Alternative  Name:  Random  VDEA  Score  1 

Measure  Name 

Measurement 

OPERATIONAL  NEEDS 

0.8 

THREAT  DETECTION 

No 

THREAT  ASSESSMENT 

No 

WARNING  PLAN 

No 

TECHNOLOGICAL  AVAILABILITY 

TRL  7 

ENVIRONMENTAL  IMPACT 

CONUS  and  Contingency  constraints 

MONETARY  PRACTICALITY  -  INITIAL 

>  105%  budget 

MONETARY  PRACTICALITY  - 
MAINTENANCE 

>  105%  budget 

ADAPTATION 

On-Site  acceptable  effort 

JOINT  OPERATIONS 

No 

NESI  DEVELOPMENT 

No 

NESI  EVALUATION 

No 

ACCESS 

Access  between  3  and  7  days 

PRODUCT  LOCAT ABILITY 

Less  than  5  min 

ACCESS  CONTROL 

Appropriate  protection  implemented 

DOCUMENT  PROTECTION 

Plan  exists,  all  products  controlled 

FILE  MANAGEMENT 

No  system 

FILE  FORMAT 

General  File  Fonnat 

SCALE 

Most  views 

DECOMPOSITION 

3+  Levels 

TOOL  FORMAT 

All  views 

DODAF  COMPLIANCY 

0.83 

REQUIREMENT  TRACEABILITY 

0 

INTERNAL  CONSISTENCY 

0.83 

EXTERNAL  CONSISTENCY 

0.83 

SME  EFFECTIVENESS 

No  Plan 

SME  INVOLVEMENT 

No  involvement 

SUPPORT  ABILITY  REQUIREMENTS 

No 

RELIABILITY  REQUIREMENTS 

Yes 

REDUNDANCY 

Some  systems,  multiple  redundancy 

RECOVERABILITY  REQUIREMENTS 

No 

CONNECTIONS 

0.83 

ARCHITECTURE  REDUNDANCY 

Between  0  and  1:500 

ARCHITECTURE  ECONOMY 

No 

OV  READABILITY 

0.8 

SV  READABILITY 

0.6 
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Alternative  Name:  Random  VDEA  Score  2 

Measure  Name 

Measurement 

OPERATIONAL  NEEDS 

0.01 

THREAT  DETECTION 

Yes 

THREAT  ASSESSMENT 

Yes 

WARNING  PLAN 

No 

TECHNOLOGICAL  AVAILABILITY 

TRL  4 

ENVIRONMENTAL  IMPACT 

Cannot  be  built 

MONETARY  PRACTICALITY  -  INITIAL 

>  105%  budget 

MONETARY  PRACTICALITY  - 
MAINTENANCE 

>  105%  budget 

ADAPTATION 

On-Site  acceptable  effort 

JOINT  OPERATIONS 

Yes 

NESI  DEVELOPMENT 

No 

NESI  EVALUATION 

Yes 

ACCESS 

Access  between  3  and  7  days 

PRODUCT  LOCAT ABILITY 

Less  than  5  min 

ACCESS  CONTROL 

Appropriate  protection  implemented 

DOCUMENT  PROTECTION 

Plan  exists,  all  products  controlled 

FILE  MANAGEMENT 

No  system 

FILE  FORMAT 

General  File  Fonnat 

SCALE 

Most  views 

DECOMPOSITION 

3+  Levels 

TOOL  FORMAT 

All  views 

DODAF  COMPLIANCY 

0.83 

REQUIREMENT  TRACEABILITY 

0 

INTERNAL  CONSISTENCY 

0.83 

EXTERNAL  CONSISTENCY 

0.83 

SME  EFFECTIVENESS 

No  Plan 

SME  INVOLVEMENT 

No  involvement 

SUPPORT  ABILITY  REQUIREMENTS 

Yes 

RELIABILITY  REQUIREMENTS 

No 

REDUNDANCY 

Some  systems,  single  redundancy 

RECOVERABILITY  REQUIREMENTS 

No 

CONNECTIONS 

0.83 

ARCHITECTURE  REDUNDANCY 

Between  0  and  1:500 

ARCHITECTURE  ECONOMY 

No 

OV  READABILITY 

0.8 

SV  READABILITY 

0.6 
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Alternative  Name:  Random  VDEA  Score  3 

Measure  Name 

Measurement 

OPERATIONAL  NEEDS 

0.205 

THREAT  DETECTION 

No 

THREAT  ASSESSMENT 

No 

WARNING  PLAN 

No 

TECHNOLOGICAL  AVAILABILITY 

TRL  4 

ENVIRONMENTAL  IMPACT 

CONUS  and  Contingency  constraints 

MONETARY  PRACTICALITY  -  INITIAL 

Between  95%  and  105%  budget 

MONETARY  PRACTICALITY  - 
MAINTENANCE 

>  105%  budget 

ADAPTATION 

On-Site  acceptable  effort 

JOINT  OPERATIONS 

No 

NESI  DEVELOPMENT 

Yes 

NESI  EVALUATION 

Yes 

ACCESS 

Access  between  3  and  7  days 

PRODUCT  LOCAT ABILITY 

Less  than  5  min 

ACCESS  CONTROL 

Appropriate  protection  implemented 

DOCUMENT  PROTECTION 

Plan  exists,  all  products  controlled 

FILE  MANAGEMENT 

No  system 

FILE  FORMAT 

General  File  Fonnat 

SCALE 

Most  views 

DECOMPOSITION 

3+  Levels 

TOOL  FORMAT 

All  views 

DODAF  COMPLIANCY 

0.83 

REQUIREMENT  TRACEABILITY 

0 

INTERNAL  CONSISTENCY 

0.83 

EXTERNAL  CONSISTENCY 

0.83 

SME  EFFECTIVENESS 

No  Plan 

SME  INVOLVEMENT 

No  involvement 

SUPPORT  ABILITY  REQUIREMENTS 

No 

RELIABILITY  REQUIREMENTS 

Yes 

REDUNDANCY 

All  systems,  multiple  redundancy 

RECOVERABILITY  REQUIREMENTS 

No 

CONNECTIONS 

0.83 

ARCHITECTURE  REDUNDANCY 

Between  0  and  1:500 

ARCHITECTURE  ECONOMY 

No 

OV  READABILITY 

0.8 

SV  READABILITY 

0.6 
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Alternative  Name:  Random  VDEA  Score  4 

Measure  Name 

Measurement 

OPERATIONAL  NEEDS 

0.866 

THREAT  DETECTION 

Yes 

THREAT  ASSESSMENT 

Yes 

WARNING  PLAN 

Yes 

TECHNOLOGICAL  AVAILABILITY 

TRL  1 

ENVIRONMENTAL  IMPACT 

CONUS,  Contingency,  and  Host  Nation  constraints 

MONETARY  PRACTICALITY  -  INITIAL 

Cost  Unknown 

MONETARY  PRACTICALITY  - 
MAINTENANCE 

Between  95%  and  105%  budget 

ADAPTATION 

Minimal  effort 

JOINT  OPERATIONS 

No 

NESI  DEVELOPMENT 

Yes 

NESI  EVALUATION 

No 

ACCESS 

Access  between  3  and  7  days 

PRODUCT  LOCAT ABILITY 

Less  than  5  min 

ACCESS  CONTROL 

Appropriate  protection  implemented 

DOCUMENT  PROTECTION 

Plan  exists,  all  products  controlled 

FILE  MANAGEMENT 

No  system 

FILE  FORMAT 

General  File  Fonnat 

SCALE 

Most  views 

DECOMPOSITION 

3+  Levels 

TOOL  FORMAT 

All  views 

DODAF  COMPLIANCY 

0.83 

REQUIREMENT  TRACEABILITY 

0 

INTERNAL  CONSISTENCY 

0.83 

EXTERNAL  CONSISTENCY 

0.83 

SME  EFFECTIVENESS 

No  Plan 

SME  INVOLVEMENT 

No  involvement 

SUPPORT  ABILITY  REQUIREMENTS 

No 

RELIABILITY  REQUIREMENTS 

Yes 

REDUNDANCY 

Some  systems,  multiple  redundancy 

RECOVERABILITY  REQUIREMENTS 

Yes 

CONNECTIONS 

0.83 

ARCHITECTURE  REDUNDANCY 

Between  0  and  1:500 

ARCHITECTURE  ECONOMY 

No 

OV  READABILITY 

0.8 

SV  READABILITY 

0.6 
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Alternative  Name:  Current  System  plus  AV-1,  SV-7,  and  SV-5 

Measure  Name 

Measurement 

OPERATIONAL  NEEDS 

1 

THREAT  DETECTION 

Yes 

THREAT  ASSESSMENT 

Yes 

WARNING  PLAN 

Yes 

TECHNOLOGICAL  AVAILABILITY 

TRL  1 

ENVIRONMENTAL  IMPACT 

CONUS  and  Contingency  constraints 

MONETARY  PRACTICALITY  -  INITIAL 

Cost  Unknown 

MONETARY  PRACTICALITY  - 
MAINTENANCE 

Cost  unknown 

ADAPTATION 

Static 

JOINT  OPERATIONS 

Yes 

NESI  DEVELOPMENT 

Yes 

NESI  EVALUATION 

No 

ACCESS 

Access  between  3  and  7  days 

PRODUCT  LOCAT ABILITY 

Less  than  5  min 

ACCESS  CONTROL 

Appropriate  protection  implemented 

DOCUMENT  PROTECTION 

Plan  exists,  all  products  controlled 

FILE  MANAGEMENT 

No  system 

FILE  FORMAT 

General  File  Fonnat 

SCALE 

Most  views 

DECOMPOSITION 

3+  Levels 

TOOL  FORMAT 

All  views 

DODAF  COMPLIANCY 

0.83 

REQUIREMENT  TRACEABILITY 

0 

INTERNAL  CONSISTENCY 

0.83 

EXTERNAL  CONSISTENCY 

0.83 

SME  EFFECTIVENESS 

No  Plan 

SME  INVOLVEMENT 

No  involvement 

SUPPORT  ABILITY  REQUIREMENTS 

Yes 

RELIABILITY  REQUIREMENTS 

No 

REDUNDANCY 

Some  systems,  single  redundancy 

RECOVERABILITY  REQUIREMENTS 

Yes 

CONNECTIONS 

0.83 

ARCHITECTURE  REDUNDANCY 

Between  0  and  1:500 

ARCHITECTURE  ECONOMY 

No 

OV  READABILITY 

0.8 

SV  READABILITY 

0.6 
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Alternative  Name:  Current  System  plus  SV-7,  SV-8  and  SV-9 

Measure  Name 

Measurement 

OPERATIONAL  NEEDS 

0 

THREAT  DETECTION 

Yes 

THREAT  ASSESSMENT 

Yes 

WARNING  PLAN 

Yes 

TECHNOLOGICAL  AVAILABILITY 

TRL  9 

ENVIRONMENTAL  IMPACT 

CONUS  and  Contingency  constraints 

MONETARY  PRACTICALITY  -  INITIAL 

Cost  Unknown 

MONETARY  PRACTICALITY  - 
MAINTENANCE 

Cost  unknown 

ADAPTATION 

Minimal  Effort 

JOINT  OPERATIONS 

Yes 

NESI  DEVELOPMENT 

Yes 

NESI  EVALUATION 

No 

ACCESS 

Access  between  3  and  7  days 

PRODUCT  LOCAT ABILITY 

Less  than  5  min 

ACCESS  CONTROL 

Appropriate  protection  implemented 

DOCUMENT  PROTECTION 

Plan  exists,  all  products  controlled 

FILE  MANAGEMENT 

No  system 

FILE  FORMAT 

General  File  Fonnat 

SCALE 

Most  views 

DECOMPOSITION 

3+  Levels 

TOOL  FORMAT 

All  views 

DODAF  COMPLIANCY 

0.83 

REQUIREMENT  TRACEABILITY 

0 

INTERNAL  CONSISTENCY 

0.83 

EXTERNAL  CONSISTENCY 

0.83 

SME  EFFECTIVENESS 

No  Plan 

SME  INVOLVEMENT 

No  involvement 

SUPPORT  ABILITY  REQUIREMENTS 

Yes 

RELIABILITY  REQUIREMENTS 

Yes 

REDUNDANCY 

Some  systems,  single  redundancy 

RECOVERABILITY  REQUIREMENTS 

Yes 

CONNECTIONS 

0.83 

ARCHITECTURE  REDUNDANCY 

Between  0  and  1:500 

ARCHITECTURE  ECONOMY 

No 

OV  READABILITY 

0.8 

SV  READABILITY 

0.6 
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